[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:06:41 dist-test-slave-grqt krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:06:41 dist-test-slave-grqt krb5kdc[399](info): set up 2 sockets
Jun 20 00:06:41 dist-test-slave-grqt krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 00:06:42.761871 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:06:42 dist-test-slave-grqt krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:06:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:06:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:06:42.880308 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36325
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36325
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:06:42.960705 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:06:42.960917 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:06:42.960940 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:06:42.962054 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:06:42.962093 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:06:42.962108 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:06:42.962121 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:06:42.962133 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:06:42.963527 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:06:42.963733 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:06:42.963919 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:06:42.969542 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:06:42.969537 412 server_base.cc:1034] running on GCE node
W20260620 00:06:42.969559 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:06:42.969559 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:06:42.970096 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:06:42.970412 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:06:42.971551 412 hybrid_clock.cc:648] HybridClock initialized: now 1781914002971516 us; error 53 us; skew 500 ppm
Jun 20 00:06:42 dist-test-slave-grqt krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:06:42.978986 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:06:42.979535 412 webserver.cc:460] Webserver started at http://127.0.98.62:46845/ using document root <none> and password file <none>
I20260620 00:06:42.979789 412 fs_manager.cc:359] Metadata directory not provided
I20260620 00:06:42.979838 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:06:42.979991 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:06:42.981118 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "c68c0b04ea1944f2b8c782994b51b397"
format_stamp: "Formatted at 2026-06-20 00:06:42 on dist-test-slave-grqt"
server_key: "d02928aa744fc318f7bfeca326fc44ba"
server_key_iv: "42d5ef79efac5de4c4e8a0e7be17ec26"
server_key_version: "encryptionkey@0"
I20260620 00:06:42.981521 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "c68c0b04ea1944f2b8c782994b51b397"
format_stamp: "Formatted at 2026-06-20 00:06:42 on dist-test-slave-grqt"
server_key: "d02928aa744fc318f7bfeca326fc44ba"
server_key_iv: "42d5ef79efac5de4c4e8a0e7be17ec26"
server_key_version: "encryptionkey@0"
I20260620 00:06:42.982954 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:06:42.983723 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:42.983918 412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:06:42.984025 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "c68c0b04ea1944f2b8c782994b51b397"
format_stamp: "Formatted at 2026-06-20 00:06:42 on dist-test-slave-grqt"
server_key: "d02928aa744fc318f7bfeca326fc44ba"
server_key_iv: "42d5ef79efac5de4c4e8a0e7be17ec26"
server_key_version: "encryptionkey@0"
I20260620 00:06:42.984097 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:06:43.006316 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:06:43.008373 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:06:43.008551 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:06:43.012771 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36325
I20260620 00:06:43.012830 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36325 every 8 connection(s)
I20260620 00:06:43.013171 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:06:43.014110 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:06:43.015271 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 412
I20260620 00:06:43.015383 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:06:43.016011 392 external_mini_cluster.cc:1428] Setting key fa0302805e65e932dd95c6890cd66e90
I20260620 00:06:43.016606 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c68c0b04ea1944f2b8c782994b51b397: Bootstrap starting.
I20260620 00:06:43.017313 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c68c0b04ea1944f2b8c782994b51b397: Neither blocks nor log segments found. Creating new log.
I20260620 00:06:43.017637 480 log.cc:824] T 00000000000000000000000000000000 P c68c0b04ea1944f2b8c782994b51b397: Log is configured to *not* fsync() on all Append() calls
I20260620 00:06:43.018293 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c68c0b04ea1944f2b8c782994b51b397: No bootstrap required, opened a new log
I20260620 00:06:43.019922 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P c68c0b04ea1944f2b8c782994b51b397 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c68c0b04ea1944f2b8c782994b51b397" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36325 } }
I20260620 00:06:43.020087 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P c68c0b04ea1944f2b8c782994b51b397 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:06:43.020131 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P c68c0b04ea1944f2b8c782994b51b397 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c68c0b04ea1944f2b8c782994b51b397, State: Initialized, Role: FOLLOWER
I20260620 00:06:43.020267 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P c68c0b04ea1944f2b8c782994b51b397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c68c0b04ea1944f2b8c782994b51b397" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36325 } }
I20260620 00:06:43.020352 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P c68c0b04ea1944f2b8c782994b51b397 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:06:43.020395 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P c68c0b04ea1944f2b8c782994b51b397 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:06:43.020445 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c68c0b04ea1944f2b8c782994b51b397 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:06:43.021296 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P c68c0b04ea1944f2b8c782994b51b397 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c68c0b04ea1944f2b8c782994b51b397" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36325 } }
I20260620 00:06:43.021432 480 leader_election.cc:302] T 00000000000000000000000000000000 P c68c0b04ea1944f2b8c782994b51b397 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c68c0b04ea1944f2b8c782994b51b397; no voters:
I20260620 00:06:43.021610 480 leader_election.cc:288] T 00000000000000000000000000000000 P c68c0b04ea1944f2b8c782994b51b397 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:06:43.021653 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c68c0b04ea1944f2b8c782994b51b397 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:06:43.021801 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P c68c0b04ea1944f2b8c782994b51b397 [term 1 LEADER]: Becoming Leader. State: Replica: c68c0b04ea1944f2b8c782994b51b397, State: Running, Role: LEADER
I20260620 00:06:43.021863 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P c68c0b04ea1944f2b8c782994b51b397 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:06:43.021891 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P c68c0b04ea1944f2b8c782994b51b397 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c68c0b04ea1944f2b8c782994b51b397" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36325 } }
I20260620 00:06:43.022245 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P c68c0b04ea1944f2b8c782994b51b397 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c68c0b04ea1944f2b8c782994b51b397. Latest consensus state: current_term: 1 leader_uuid: "c68c0b04ea1944f2b8c782994b51b397" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c68c0b04ea1944f2b8c782994b51b397" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36325 } } }
I20260620 00:06:43.022325 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P c68c0b04ea1944f2b8c782994b51b397 [sys.catalog]: This master's current role is: LEADER
I20260620 00:06:43.022255 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P c68c0b04ea1944f2b8c782994b51b397 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c68c0b04ea1944f2b8c782994b51b397" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c68c0b04ea1944f2b8c782994b51b397" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36325 } } }
I20260620 00:06:43.022547 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P c68c0b04ea1944f2b8c782994b51b397 [sys.catalog]: This master's current role is: LEADER
W20260620 00:06:43.023684 501 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c68c0b04ea1944f2b8c782994b51b397: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:06:43.023741 501 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:06:43.024540 491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:06:43.024729 491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:06:43.026320 491 catalog_manager.cc:1347] Generated new cluster ID: 2136be236c0c4379bdaf159253c34ea3
I20260620 00:06:43.026371 491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 00:06:43 dist-test-slave-grqt krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:06:43.032709 491 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:06:43.033424 491 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:06:43.033691 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:43.016888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42486 (local address 127.0.98.62:36325)
0620 00:06:43.017118 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:43.017128 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:43.017300 (+ 172us) server_negotiation.cc:407] Connection header received
0620 00:06:43.019901 (+ 2601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:43.019907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:43.019982 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:43.020126 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:43.021954 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:43.023281 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:43.024453 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:43.025034 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:43.025145 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:43.029412 (+ 4267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:43.029416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:43.029417 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:43.029428 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:43.031398 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:43.031927 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:43.031930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:43.031930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:43.032049 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:43.032340 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:43.032341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:43.032342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:43.032624 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:43.032813 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:43.033165 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 00:06:43.033351 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":185,"thread_start_us":119,"threads_started":1}
I20260620 00:06:43.042160 491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c68c0b04ea1944f2b8c782994b51b397: Generated new TSK 0
I20260620 00:06:43.042328 491 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:06:43.073247 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36325
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:06:43.156864 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:06:43.157102 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:06:43.157128 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:06:43.158409 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:06:43.158463 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:06:43.158492 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:06:43.160039 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36325
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:06:43.160288 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:06:43.160555 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:06:43.165987 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:06:43.165990 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:06:43.166028 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:06:43.166019 506 server_base.cc:1034] running on GCE node
I20260620 00:06:43.166472 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:06:43.166742 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:06:43.167917 506 hybrid_clock.cc:648] HybridClock initialized: now 1781914003167900 us; error 35 us; skew 500 ppm
Jun 20 00:06:43 dist-test-slave-grqt krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914003, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:06:43.170554 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:06:43.171058 506 webserver.cc:460] Webserver started at http://127.0.98.1:37067/ using document root <none> and password file <none>
I20260620 00:06:43.171283 506 fs_manager.cc:359] Metadata directory not provided
I20260620 00:06:43.171327 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:06:43.171499 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:06:43.172562 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-0/data/instance:
uuid: "0ede68b46f824ef3b81d465cd25f1a28"
format_stamp: "Formatted at 2026-06-20 00:06:43 on dist-test-slave-grqt"
server_key: "48b927b56397dc3b4e1fc101cf3dc9a2"
server_key_iv: "55c8fecf8086c061c91673512ffaaf44"
server_key_version: "encryptionkey@0"
I20260620 00:06:43.172932 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0ede68b46f824ef3b81d465cd25f1a28"
format_stamp: "Formatted at 2026-06-20 00:06:43 on dist-test-slave-grqt"
server_key: "48b927b56397dc3b4e1fc101cf3dc9a2"
server_key_iv: "55c8fecf8086c061c91673512ffaaf44"
server_key_version: "encryptionkey@0"
I20260620 00:06:43.174387 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:06:43.175102 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:43.175287 506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 00:06:43.175379 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-0/wal
uuid: "0ede68b46f824ef3b81d465cd25f1a28"
format_stamp: "Formatted at 2026-06-20 00:06:43 on dist-test-slave-grqt"
server_key: "48b927b56397dc3b4e1fc101cf3dc9a2"
server_key_iv: "55c8fecf8086c061c91673512ffaaf44"
server_key_version: "encryptionkey@0"
I20260620 00:06:43.175452 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:06:43.194792 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:06:43.197057 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:06:43.197222 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:06:43.197468 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:06:43.197772 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:06:43.197824 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:43.197865 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:06:43.197891 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:43.203766 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46513
I20260620 00:06:43.203845 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46513 every 8 connection(s)
I20260620 00:06:43.204186 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:06:43.208436 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 506
I20260620 00:06:43.208561 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:06:43.208724 392 external_mini_cluster.cc:1428] Setting key 62930d9f49bdf6116435eb2be517e388
Jun 20 00:06:43 dist-test-slave-grqt krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914003, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:06:43.215009 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:43.205026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58001 (local address 127.0.98.62:36325)
0620 00:06:43.205107 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:43.205125 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:43.205331 (+ 206us) server_negotiation.cc:407] Connection header received
0620 00:06:43.205945 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:43.205946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:43.205959 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:43.205999 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:43.207104 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:43.207626 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:43.208767 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:43.209079 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:43.209159 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:43.211574 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:43.211576 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:43.211576 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:43.211584 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:43.213439 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:43.213905 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:43.213907 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:43.213908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:43.213990 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:43.214238 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:43.214239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:43.214240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:43.214405 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:43.214487 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:43.214756 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 00:06:43.214868 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 00:06:43.215054 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:43.205027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36325 (local address 127.0.98.1:58001)
0620 00:06:43.205197 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:43.205242 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:43.205816 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:43.206093 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:43.206098 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:43.206716 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:06:43.206921 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:43.206926 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:43.207752 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:43.207753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:43.208667 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:43.208670 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:43.209166 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:43.209167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:43.209314 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:43.209981 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:06:43.209989 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:06:43.211490 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:06:43.213552 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:43.213560 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:43.213562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:43.213814 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:43.214065 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:43.214066 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:43.214066 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:43.214169 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:43.214479 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:06:43.214483 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:06:43.214617 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 00:06:43.214704 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 00:06:43.214846 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":70,"threads_started":1}
I20260620 00:06:43.215938 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36325
I20260620 00:06:43.216068 635 heartbeater.cc:464] Registering TS with master...
I20260620 00:06:43.216256 635 heartbeater.cc:511] Master 127.0.98.62:36325 requested a full tablet report, sending...
I20260620 00:06:43.216799 444 ts_manager.cc:194] Registered new tserver with Master: 0ede68b46f824ef3b81d465cd25f1a28 (127.0.98.1:46513)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 00:06:43.217528 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58001
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:06:43.236946 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36325
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:06:43.318046 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:06:43.318276 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:06:43.318302 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:06:43.319550 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:06:43.319598 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:06:43.319628 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:06:43.321106 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36325
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:06:43.321385 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:06:43.321627 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:06:43.326990 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:06:43.327029 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:06:43.327071 640 server_base.cc:1034] running on GCE node
W20260620 00:06:43.326999 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:06:43.327409 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:06:43.327633 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:06:43.328770 640 hybrid_clock.cc:648] HybridClock initialized: now 1781914003328760 us; error 27 us; skew 500 ppm
Jun 20 00:06:43 dist-test-slave-grqt krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914003, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:06:43.331301 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:06:43.331692 640 webserver.cc:460] Webserver started at http://127.0.98.2:44229/ using document root <none> and password file <none>
I20260620 00:06:43.331872 640 fs_manager.cc:359] Metadata directory not provided
I20260620 00:06:43.331912 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:06:43.331998 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:06:43.333002 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-1/data/instance:
uuid: "c838528a120a489091e0425fb41affe9"
format_stamp: "Formatted at 2026-06-20 00:06:43 on dist-test-slave-grqt"
server_key: "ec3ed37ac86958344ac8c257e68e35d1"
server_key_iv: "8059a63af3ee75724e8ec0ced24200f1"
server_key_version: "encryptionkey@0"
I20260620 00:06:43.333313 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c838528a120a489091e0425fb41affe9"
format_stamp: "Formatted at 2026-06-20 00:06:43 on dist-test-slave-grqt"
server_key: "ec3ed37ac86958344ac8c257e68e35d1"
server_key_iv: "8059a63af3ee75724e8ec0ced24200f1"
server_key_version: "encryptionkey@0"
I20260620 00:06:43.334714 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:06:43.335526 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:43.335729 640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:06:43.335822 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-1/wal
uuid: "c838528a120a489091e0425fb41affe9"
format_stamp: "Formatted at 2026-06-20 00:06:43 on dist-test-slave-grqt"
server_key: "ec3ed37ac86958344ac8c257e68e35d1"
server_key_iv: "8059a63af3ee75724e8ec0ced24200f1"
server_key_version: "encryptionkey@0"
I20260620 00:06:43.335880 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:06:43.355515 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:06:43.357873 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:06:43.357990 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:06:43.358191 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:06:43.358440 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:06:43.358471 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:43.358493 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:06:43.358506 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:43.365306 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33499
I20260620 00:06:43.365357 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33499 every 8 connection(s)
I20260620 00:06:43.365707 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 00:06:43 dist-test-slave-grqt krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914003, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:06:43.371960 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 640
I20260620 00:06:43.372081 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:06:43.372251 392 external_mini_cluster.cc:1428] Setting key c614f950e243721e60e2e87dcca41ffb
I20260620 00:06:43.376166 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:43.366434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57697 (local address 127.0.98.62:36325)
0620 00:06:43.366510 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:43.366512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:43.366736 (+ 224us) server_negotiation.cc:407] Connection header received
0620 00:06:43.367251 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:43.367253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:43.367263 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:43.367335 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:43.368250 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:43.368734 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:43.369798 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:43.369983 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:43.370045 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:43.372536 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:43.372538 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:43.372538 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:43.372546 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:43.374500 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:43.375153 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:43.375154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:43.375154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:43.375212 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:43.375536 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:43.375538 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:43.375538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:43.375677 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:43.375745 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:43.376034 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 00:06:43.376102 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 00:06:43.376410 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:43.366393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36325 (local address 127.0.98.2:57697)
0620 00:06:43.366580 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:43.366625 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:43.367129 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:43.367393 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:43.367396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:43.367966 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:06:43.368165 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:43.368169 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:43.368815 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:43.368816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:43.369722 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:43.369724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:43.370059 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:43.370060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:43.370156 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:43.370683 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:06:43.370693 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:06:43.372401 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:06:43.374625 (+ 2224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:43.374635 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:43.374638 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:43.375061 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:43.375284 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:43.375285 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:43.375285 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:43.375439 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:43.375758 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:06:43.375765 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:06:43.375911 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 00:06:43.376004 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 00:06:43.376136 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":79,"threads_started":1}
I20260620 00:06:43.376865 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36325
I20260620 00:06:43.376996 769 heartbeater.cc:464] Registering TS with master...
I20260620 00:06:43.377211 769 heartbeater.cc:511] Master 127.0.98.62:36325 requested a full tablet report, sending...
I20260620 00:06:43.377717 444 ts_manager.cc:194] Registered new tserver with Master: c838528a120a489091e0425fb41affe9 (127.0.98.2:33499)
I20260620 00:06:43.378078 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57697
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:06:43.400746 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36325
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:06:43.483379 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:06:43.483623 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:06:43.483662 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:06:43.484807 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:06:43.484871 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:06:43.484915 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:06:43.486423 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36325
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:06:43.486693 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:06:43.486918 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:06:43.492110 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:06:43.492419 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:06:43.492501 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:06:43.492682 774 server_base.cc:1034] running on GCE node
I20260620 00:06:43.492874 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:06:43.493101 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:06:43.494275 774 hybrid_clock.cc:648] HybridClock initialized: now 1781914003494246 us; error 44 us; skew 500 ppm
Jun 20 00:06:43 dist-test-slave-grqt krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914003, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:06:43.497174 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:06:43.497742 774 webserver.cc:460] Webserver started at http://127.0.98.3:37215/ using document root <none> and password file <none>
I20260620 00:06:43.497989 774 fs_manager.cc:359] Metadata directory not provided
I20260620 00:06:43.498039 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:06:43.498205 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:06:43.499421 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-2/data/instance:
uuid: "fc1dd9af2753401ca09b8d489d164253"
format_stamp: "Formatted at 2026-06-20 00:06:43 on dist-test-slave-grqt"
server_key: "9238ccf5006e2e5b2cd61df40abcab7a"
server_key_iv: "47e2e91c582fa0c899ab4ff4d547be82"
server_key_version: "encryptionkey@0"
I20260620 00:06:43.499861 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fc1dd9af2753401ca09b8d489d164253"
format_stamp: "Formatted at 2026-06-20 00:06:43 on dist-test-slave-grqt"
server_key: "9238ccf5006e2e5b2cd61df40abcab7a"
server_key_iv: "47e2e91c582fa0c899ab4ff4d547be82"
server_key_version: "encryptionkey@0"
I20260620 00:06:43.501840 774 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260620 00:06:43.502691 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:43.502893 774 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:06:43.502988 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-2/wal
uuid: "fc1dd9af2753401ca09b8d489d164253"
format_stamp: "Formatted at 2026-06-20 00:06:43 on dist-test-slave-grqt"
server_key: "9238ccf5006e2e5b2cd61df40abcab7a"
server_key_iv: "47e2e91c582fa0c899ab4ff4d547be82"
server_key_version: "encryptionkey@0"
I20260620 00:06:43.503101 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:06:43.518127 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:06:43.520303 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:06:43.520510 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:06:43.520782 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:06:43.521123 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:06:43.521183 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:43.521234 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:06:43.521272 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:43.528942 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45167
I20260620 00:06:43.528982 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45167 every 8 connection(s)
I20260620 00:06:43.529345 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:06:43.535509 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 774
I20260620 00:06:43.535573 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914001714546-392-0/minicluster-data/ts-2/wal/instance
Jun 20 00:06:43 dist-test-slave-grqt krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914003, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:06:43.535734 392 external_mini_cluster.cc:1428] Setting key b812e6df2a44047106fc37de20968150
I20260620 00:06:43.539294 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:43.530141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55953 (local address 127.0.98.62:36325)
0620 00:06:43.530213 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:43.530215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:43.530438 (+ 223us) server_negotiation.cc:407] Connection header received
0620 00:06:43.530899 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:43.530901 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:43.530950 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:43.531022 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:43.531949 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:43.532491 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:43.533533 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:43.533732 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:43.533792 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:43.536130 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:43.536148 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:43.536148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:43.536157 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:43.537846 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:43.538249 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:43.538249 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:43.538250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:43.538302 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:43.538569 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:43.538570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:43.538615 (+ 45us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:43.538799 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:43.538865 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:43.539162 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 00:06:43.539230 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 00:06:43.539698 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:43.530127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36325 (local address 127.0.98.3:55953)
0620 00:06:43.530301 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:43.530343 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:43.530813 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:43.531076 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:43.531079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:43.531633 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:06:43.531832 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:43.531837 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:43.532573 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:43.532574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:43.533448 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:43.533450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:43.533814 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:43.533815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:43.534000 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:43.534516 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:06:43.534524 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:06:43.536014 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:06:43.537923 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:43.537930 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:43.537933 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:43.538179 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:43.538371 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:43.538372 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:43.538372 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:43.538487 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:43.538886 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:06:43.538891 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:06:43.539031 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 00:06:43.539128 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 00:06:43.539273 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":103,"threads_started":1}
I20260620 00:06:43.540266 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36325
I20260620 00:06:43.540369 903 heartbeater.cc:464] Registering TS with master...
I20260620 00:06:43.540566 903 heartbeater.cc:511] Master 127.0.98.62:36325 requested a full tablet report, sending...
I20260620 00:06:43.540937 444 ts_manager.cc:194] Registered new tserver with Master: fc1dd9af2753401ca09b8d489d164253 (127.0.98.3:45167)
I20260620 00:06:43.541283 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55953
I20260620 00:06:43.547920 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 00:06:43 dist-test-slave-grqt krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914003, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 00:06:43 dist-test-slave-grqt krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914003, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 00:06:43.571593 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:43.563761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38942 (local address 127.0.98.1:46513)
0620 00:06:43.563920 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:43.563927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:43.563938 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:06:43.563974 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:43.563977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:43.563991 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:43.564077 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:43.564653 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:43.565431 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:43.566264 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:43.566592 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:43.566672 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:43.568848 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:43.568861 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:43.568861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:43.568868 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:43.570369 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:43.570770 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:43.570774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:43.570774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:43.570833 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:43.571033 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:43.571034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:43.571034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:43.571172 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:43.571323 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:43.571371 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 00:06:43.571461 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":60,"threads_started":1}
W20260620 00:06:43.571974 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38942
Jun 20 00:06:43 dist-test-slave-grqt krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 00:06:43 dist-test-slave-grqt krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 00:06:43.593775 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:43.586735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38944 (local address 127.0.98.1:46513)
0620 00:06:43.586836 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:43.586839 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:43.586965 (+ 126us) server_negotiation.cc:407] Connection header received
0620 00:06:43.587044 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:43.587045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:43.587063 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:43.587126 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:43.587715 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:43.588246 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:43.588943 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:43.589145 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:43.589198 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:43.591444 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:43.591446 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:43.591446 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:43.591452 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:43.592671 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:43.593010 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:43.593012 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:43.593012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:43.593091 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:43.593270 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:43.593270 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:43.593270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:43.593417 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:43.593516 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:43.593631 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 00:06:43.593701 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 00:06:43.594343 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 506
I20260620 00:06:43.599236 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 640
I20260620 00:06:43.604769 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 774
I20260620 00:06:43.609100 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 412
2026-06-20T00:06:43Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1900 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:06:43 dist-test-slave-grqt krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:06:43 dist-test-slave-grqt krb5kdc[917](info): set up 2 sockets
Jun 20 00:06:43 dist-test-slave-grqt krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260620 00:06:44.668975 392 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:06:44 dist-test-slave-grqt krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:06:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:06:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:06:44.796713 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33831
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33831
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:06:44.879091 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:06:44.879333 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:06:44.879370 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:06:44.880522 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:06:44.880590 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:06:44.880618 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:06:44.880651 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:06:44.880673 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:06:44.882242 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:06:44.882519 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:06:44.882745 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:06:44.887706 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:06:44.887741 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:06:44.887741 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:06:44.887769 930 server_base.cc:1034] running on GCE node
I20260620 00:06:44.888162 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:06:44.888453 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:06:44.889643 930 hybrid_clock.cc:648] HybridClock initialized: now 1781914004889630 us; error 31 us; skew 500 ppm
Jun 20 00:06:44 dist-test-slave-grqt krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914004, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:06:44.892442 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:06:44.892912 930 webserver.cc:460] Webserver started at http://127.0.98.62:45223/ using document root <none> and password file <none>
I20260620 00:06:44.893142 930 fs_manager.cc:359] Metadata directory not provided
I20260620 00:06:44.893208 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:06:44.893348 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:06:44.894896 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "2c853d45ae6749b79388b2222b2a9530"
format_stamp: "Formatted at 2026-06-20 00:06:44 on dist-test-slave-grqt"
server_key: "21caffd9fddce02b84f87129852771cf"
server_key_iv: "d7df2c3abde403e559f44d4ec1978a7c"
server_key_version: "encryptionkey@0"
I20260620 00:06:44.895306 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "2c853d45ae6749b79388b2222b2a9530"
format_stamp: "Formatted at 2026-06-20 00:06:44 on dist-test-slave-grqt"
server_key: "21caffd9fddce02b84f87129852771cf"
server_key_iv: "d7df2c3abde403e559f44d4ec1978a7c"
server_key_version: "encryptionkey@0"
I20260620 00:06:44.896961 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:06:44.897879 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:44.898145 930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:06:44.898264 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "2c853d45ae6749b79388b2222b2a9530"
format_stamp: "Formatted at 2026-06-20 00:06:44 on dist-test-slave-grqt"
server_key: "21caffd9fddce02b84f87129852771cf"
server_key_iv: "d7df2c3abde403e559f44d4ec1978a7c"
server_key_version: "encryptionkey@0"
I20260620 00:06:44.898350 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:06:44.907755 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:06:44.910874 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:06:44.911110 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:06:44.915279 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33831
I20260620 00:06:44.915313 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33831 every 8 connection(s)
I20260620 00:06:44.915670 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:06:44.916551 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:06:44.918746 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c853d45ae6749b79388b2222b2a9530: Bootstrap starting.
I20260620 00:06:44.919284 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c853d45ae6749b79388b2222b2a9530: Neither blocks nor log segments found. Creating new log.
I20260620 00:06:44.919518 998 log.cc:824] T 00000000000000000000000000000000 P 2c853d45ae6749b79388b2222b2a9530: Log is configured to *not* fsync() on all Append() calls
I20260620 00:06:44.920178 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c853d45ae6749b79388b2222b2a9530: No bootstrap required, opened a new log
I20260620 00:06:44.921742 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 930
I20260620 00:06:44.921837 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:06:44.921855 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c853d45ae6749b79388b2222b2a9530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c853d45ae6749b79388b2222b2a9530" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33831 } }
I20260620 00:06:44.922009 392 external_mini_cluster.cc:1428] Setting key 0be0d5f3d7f6ca01aed25b03af0d5be5
I20260620 00:06:44.922036 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c853d45ae6749b79388b2222b2a9530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:06:44.922063 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c853d45ae6749b79388b2222b2a9530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c853d45ae6749b79388b2222b2a9530, State: Initialized, Role: FOLLOWER
I20260620 00:06:44.922226 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c853d45ae6749b79388b2222b2a9530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c853d45ae6749b79388b2222b2a9530" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33831 } }
I20260620 00:06:44.922310 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c853d45ae6749b79388b2222b2a9530 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:06:44.922346 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c853d45ae6749b79388b2222b2a9530 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:06:44.922396 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c853d45ae6749b79388b2222b2a9530 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:06:44.923300 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c853d45ae6749b79388b2222b2a9530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c853d45ae6749b79388b2222b2a9530" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33831 } }
I20260620 00:06:44.923421 998 leader_election.cc:302] T 00000000000000000000000000000000 P 2c853d45ae6749b79388b2222b2a9530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c853d45ae6749b79388b2222b2a9530; no voters:
I20260620 00:06:44.923573 998 leader_election.cc:288] T 00000000000000000000000000000000 P 2c853d45ae6749b79388b2222b2a9530 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:06:44.923604 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c853d45ae6749b79388b2222b2a9530 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:06:44.923707 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c853d45ae6749b79388b2222b2a9530 [term 1 LEADER]: Becoming Leader. State: Replica: 2c853d45ae6749b79388b2222b2a9530, State: Running, Role: LEADER
I20260620 00:06:44.923787 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c853d45ae6749b79388b2222b2a9530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c853d45ae6749b79388b2222b2a9530" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33831 } }
I20260620 00:06:44.923825 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c853d45ae6749b79388b2222b2a9530 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:06:44.924101 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c853d45ae6749b79388b2222b2a9530 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c853d45ae6749b79388b2222b2a9530" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c853d45ae6749b79388b2222b2a9530" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33831 } } }
I20260620 00:06:44.924196 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c853d45ae6749b79388b2222b2a9530 [sys.catalog]: This master's current role is: LEADER
I20260620 00:06:44.924345 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c853d45ae6749b79388b2222b2a9530 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c853d45ae6749b79388b2222b2a9530. Latest consensus state: current_term: 1 leader_uuid: "2c853d45ae6749b79388b2222b2a9530" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c853d45ae6749b79388b2222b2a9530" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33831 } } }
I20260620 00:06:44.924405 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c853d45ae6749b79388b2222b2a9530 [sys.catalog]: This master's current role is: LEADER
I20260620 00:06:44.924510 1007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:06:44.925211 1007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:06:44.927515 1007 catalog_manager.cc:1347] Generated new cluster ID: 7c6019ee81ed4fd2a37a2f3993e00cff
I20260620 00:06:44.927572 1007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 00:06:44 dist-test-slave-grqt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:06:44.933182 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:44.922520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57490 (local address 127.0.98.62:33831)
0620 00:06:44.922726 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:44.922736 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:44.922775 (+ 39us) server_negotiation.cc:407] Connection header received
0620 00:06:44.922878 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:44.922883 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:44.922924 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:44.923129 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:44.923878 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:44.925251 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:44.926217 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:44.926645 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:44.926781 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:44.929233 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:44.929237 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:44.929237 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:44.929250 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:44.931476 (+ 2226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:44.931887 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:44.931891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:44.931891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:44.932006 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:44.932251 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:44.932252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:44.932252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:44.932521 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:44.932702 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:44.932800 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 00:06:44.932963 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":100,"threads_started":1}
I20260620 00:06:44.947726 1007 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:06:44.948295 1007 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:06:44.951239 1007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c853d45ae6749b79388b2222b2a9530: Generated new TSK 0
I20260620 00:06:44.951434 1007 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:06:44.985270 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33831
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:06:45.067663 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:06:45.067938 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:06:45.067984 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:06:45.069255 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:06:45.069327 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:06:45.069375 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:06:45.070930 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33831
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:06:45.071197 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:06:45.071434 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:06:45.076618 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:06:45.076551 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:06:45.076541 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:06:45.076771 1024 server_base.cc:1034] running on GCE node
I20260620 00:06:45.076982 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:06:45.077301 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:06:45.078495 1024 hybrid_clock.cc:648] HybridClock initialized: now 1781914005078471 us; error 44 us; skew 500 ppm
Jun 20 00:06:45 dist-test-slave-grqt krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:06:45.080989 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:06:45.081513 1024 webserver.cc:460] Webserver started at http://127.0.98.1:42957/ using document root <none> and password file <none>
I20260620 00:06:45.081787 1024 fs_manager.cc:359] Metadata directory not provided
I20260620 00:06:45.081845 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:06:45.081940 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:06:45.082947 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-0/data/instance:
uuid: "ea54912757f743de96d4629a7f97f37c"
format_stamp: "Formatted at 2026-06-20 00:06:45 on dist-test-slave-grqt"
server_key: "aeecc0cbd79812f818e2252d08d2b638"
server_key_iv: "e877a852256d582e6337f91109aaf4d2"
server_key_version: "encryptionkey@0"
I20260620 00:06:45.083247 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ea54912757f743de96d4629a7f97f37c"
format_stamp: "Formatted at 2026-06-20 00:06:45 on dist-test-slave-grqt"
server_key: "aeecc0cbd79812f818e2252d08d2b638"
server_key_iv: "e877a852256d582e6337f91109aaf4d2"
server_key_version: "encryptionkey@0"
I20260620 00:06:45.084501 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:06:45.085222 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:45.085469 1024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:06:45.085574 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-0/wal
uuid: "ea54912757f743de96d4629a7f97f37c"
format_stamp: "Formatted at 2026-06-20 00:06:45 on dist-test-slave-grqt"
server_key: "aeecc0cbd79812f818e2252d08d2b638"
server_key_iv: "e877a852256d582e6337f91109aaf4d2"
server_key_version: "encryptionkey@0"
I20260620 00:06:45.085649 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:06:45.094913 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:06:45.096860 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:06:45.097018 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:06:45.097277 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:06:45.097630 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:06:45.097684 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:45.097726 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:06:45.097755 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:45.104101 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40705
I20260620 00:06:45.104148 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40705 every 8 connection(s)
I20260620 00:06:45.104476 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 00:06:45 dist-test-slave-grqt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:06:45.110838 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 1024
I20260620 00:06:45.110939 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:06:45.111135 392 external_mini_cluster.cc:1428] Setting key 84c6eae1fdb238d232c80f0722f89c12
I20260620 00:06:45.114161 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:45.105197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57093 (local address 127.0.98.62:33831)
0620 00:06:45.105277 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:45.105279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:45.105535 (+ 256us) server_negotiation.cc:407] Connection header received
0620 00:06:45.106049 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:45.106050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:45.106061 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:45.106107 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:45.107029 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:45.107528 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:45.108519 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:45.108714 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:45.108767 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:45.110860 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:45.110861 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:45.110862 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:45.110872 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:45.112719 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:45.113134 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:45.113136 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:45.113136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:45.113201 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:45.113503 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:45.113504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:45.113505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:45.113627 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:45.113706 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:45.113981 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 00:06:45.114068 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 00:06:45.114400 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:45.105176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33831 (local address 127.0.98.1:57093)
0620 00:06:45.105398 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:45.105449 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:45.105960 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:45.106160 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:45.106164 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:45.106721 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:06:45.106920 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:45.106924 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:45.107612 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:45.107613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:45.108444 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:45.108446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:45.108783 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:45.108784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:45.108879 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:45.109355 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:06:45.109416 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:06:45.110756 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:06:45.112810 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:45.112816 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:45.112818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:45.113036 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:45.113269 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:45.113270 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:45.113270 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:45.113437 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:45.113696 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:06:45.113702 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:06:45.113846 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 00:06:45.113949 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 00:06:45.114120 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":106,"threads_started":1}
I20260620 00:06:45.114966 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33831
I20260620 00:06:45.115083 1153 heartbeater.cc:464] Registering TS with master...
I20260620 00:06:45.115289 1153 heartbeater.cc:511] Master 127.0.98.62:33831 requested a full tablet report, sending...
I20260620 00:06:45.115847 962 ts_manager.cc:194] Registered new tserver with Master: ea54912757f743de96d4629a7f97f37c (127.0.98.1:40705)
I20260620 00:06:45.116847 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57093
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:06:45.136905 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33831
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:06:45.218715 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:06:45.218943 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:06:45.218968 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:06:45.220191 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:06:45.220238 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:06:45.220268 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:06:45.221815 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33831
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:06:45.222081 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:06:45.222321 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:06:45.227794 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:06:45.227802 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:06:45.227870 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:06:45.228061 1158 server_base.cc:1034] running on GCE node
I20260620 00:06:45.228268 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:06:45.228516 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:06:45.229676 1158 hybrid_clock.cc:648] HybridClock initialized: now 1781914005229654 us; error 43 us; skew 500 ppm
Jun 20 00:06:45 dist-test-slave-grqt krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:06:45.232231 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:06:45.232656 1158 webserver.cc:460] Webserver started at http://127.0.98.2:40335/ using document root <none> and password file <none>
I20260620 00:06:45.232885 1158 fs_manager.cc:359] Metadata directory not provided
I20260620 00:06:45.232951 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:06:45.233100 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:06:45.234216 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-1/data/instance:
uuid: "565c0bec474d4984801748f4d3441570"
format_stamp: "Formatted at 2026-06-20 00:06:45 on dist-test-slave-grqt"
server_key: "17cea8948277076009333a5e86a09e05"
server_key_iv: "9336733d9c9d7079ce0195b62f84bb42"
server_key_version: "encryptionkey@0"
I20260620 00:06:45.234565 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-1/wal/instance:
uuid: "565c0bec474d4984801748f4d3441570"
format_stamp: "Formatted at 2026-06-20 00:06:45 on dist-test-slave-grqt"
server_key: "17cea8948277076009333a5e86a09e05"
server_key_iv: "9336733d9c9d7079ce0195b62f84bb42"
server_key_version: "encryptionkey@0"
I20260620 00:06:45.235857 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 00:06:45.236608 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:45.236776 1158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 00:06:45.236874 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-1/wal
uuid: "565c0bec474d4984801748f4d3441570"
format_stamp: "Formatted at 2026-06-20 00:06:45 on dist-test-slave-grqt"
server_key: "17cea8948277076009333a5e86a09e05"
server_key_iv: "9336733d9c9d7079ce0195b62f84bb42"
server_key_version: "encryptionkey@0"
I20260620 00:06:45.236954 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:06:45.259972 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:06:45.262470 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:06:45.262601 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:06:45.262790 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:06:45.263106 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:06:45.263140 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:45.263185 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:06:45.263213 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:45.268939 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46227
I20260620 00:06:45.269018 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46227 every 8 connection(s)
I20260620 00:06:45.269351 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:06:45.271782 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 1158
I20260620 00:06:45.271879 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:06:45.272051 392 external_mini_cluster.cc:1428] Setting key 3de482bea85d2d4a23191074ac8ab42f
Jun 20 00:06:45 dist-test-slave-grqt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:06:45.279557 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:45.270172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53397 (local address 127.0.98.62:33831)
0620 00:06:45.270231 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:45.270233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:45.270478 (+ 245us) server_negotiation.cc:407] Connection header received
0620 00:06:45.271084 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:45.271085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:45.271101 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:45.271171 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:45.272382 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:45.272985 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:45.274113 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:45.274313 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:45.274389 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:45.276619 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:45.276621 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:45.276622 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:45.276630 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:45.278111 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:45.278531 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:45.278532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:45.278533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:45.278593 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:45.278837 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:45.278838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:45.278838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:45.278954 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:45.279096 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:45.279370 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 00:06:45.279463 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 00:06:45.279768 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:45.270124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33831 (local address 127.0.98.2:53397)
0620 00:06:45.270290 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:45.270347 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:45.270843 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:45.271249 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:45.271253 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:45.272000 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:06:45.272301 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:45.272309 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:45.273090 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:45.273091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:45.274013 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:45.274015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:45.274394 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:45.274396 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:45.274535 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:45.275121 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:06:45.275130 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:06:45.276518 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:06:45.278199 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:45.278206 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:45.278209 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:45.278433 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:45.278660 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:45.278661 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:45.278661 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:45.278768 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:45.279088 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:06:45.279095 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:06:45.279238 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 00:06:45.279352 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 00:06:45.279527 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 00:06:45.280212 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33831
I20260620 00:06:45.280308 1287 heartbeater.cc:464] Registering TS with master...
I20260620 00:06:45.280506 1287 heartbeater.cc:511] Master 127.0.98.62:33831 requested a full tablet report, sending...
I20260620 00:06:45.280896 962 ts_manager.cc:194] Registered new tserver with Master: 565c0bec474d4984801748f4d3441570 (127.0.98.2:46227)
I20260620 00:06:45.281343 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53397
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:06:45.300103 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33831
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:06:45.383731 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:06:45.383953 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:06:45.383978 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:06:45.385175 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:06:45.385219 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:06:45.385246 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:06:45.386705 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33831
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:06:45.386968 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:06:45.387233 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:06:45.392412 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:06:45.392454 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:06:45.392550 1292 server_base.cc:1034] running on GCE node
W20260620 00:06:45.392422 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:06:45.392904 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:06:45.393199 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:06:45.394397 1292 hybrid_clock.cc:648] HybridClock initialized: now 1781914005394349 us; error 64 us; skew 500 ppm
Jun 20 00:06:45 dist-test-slave-grqt krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:06:45.396967 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:06:45.397480 1292 webserver.cc:460] Webserver started at http://127.0.98.3:39331/ using document root <none> and password file <none>
I20260620 00:06:45.397747 1292 fs_manager.cc:359] Metadata directory not provided
I20260620 00:06:45.397816 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:06:45.397974 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:06:45.399066 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-2/data/instance:
uuid: "5006d1a946bc4ff2baca9653516b6b31"
format_stamp: "Formatted at 2026-06-20 00:06:45 on dist-test-slave-grqt"
server_key: "2b635cb2cd798f32b769173cc5aa901e"
server_key_iv: "3233551aab4ecbb23a2d7aac162029b4"
server_key_version: "encryptionkey@0"
I20260620 00:06:45.399441 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5006d1a946bc4ff2baca9653516b6b31"
format_stamp: "Formatted at 2026-06-20 00:06:45 on dist-test-slave-grqt"
server_key: "2b635cb2cd798f32b769173cc5aa901e"
server_key_iv: "3233551aab4ecbb23a2d7aac162029b4"
server_key_version: "encryptionkey@0"
I20260620 00:06:45.400799 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:06:45.401754 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:45.401940 1292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:06:45.402117 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-2/wal
uuid: "5006d1a946bc4ff2baca9653516b6b31"
format_stamp: "Formatted at 2026-06-20 00:06:45 on dist-test-slave-grqt"
server_key: "2b635cb2cd798f32b769173cc5aa901e"
server_key_iv: "3233551aab4ecbb23a2d7aac162029b4"
server_key_version: "encryptionkey@0"
I20260620 00:06:45.402207 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:06:45.415587 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:06:45.417582 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:06:45.417701 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:06:45.417884 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:06:45.418218 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:06:45.418254 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:45.418301 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:06:45.418329 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:45.424706 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39795
I20260620 00:06:45.424742 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39795 every 8 connection(s)
I20260620 00:06:45.425180 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 00:06:45 dist-test-slave-grqt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:06:45.435364 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 1292
I20260620 00:06:45.435477 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914001714546-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:06:45.435660 392 external_mini_cluster.cc:1428] Setting key 01497698e753a5189d433d16ef80ba34
I20260620 00:06:45.435959 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:45.426003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45179 (local address 127.0.98.62:33831)
0620 00:06:45.426077 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:45.426078 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:45.426319 (+ 241us) server_negotiation.cc:407] Connection header received
0620 00:06:45.426833 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:45.426834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:45.426845 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:45.426918 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:45.427930 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:45.428482 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:45.429686 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:45.429881 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:45.429959 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:45.432311 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:45.432313 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:45.432313 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:45.432322 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:45.434032 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:45.434601 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:45.434603 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:45.434603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:45.434723 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:45.435221 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:45.435222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:45.435222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:45.435354 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:45.435439 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:45.435744 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 00:06:45.435869 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 00:06:45.436160 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:45.425995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33831 (local address 127.0.98.3:45179)
0620 00:06:45.426174 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:45.426218 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:45.426737 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:45.427020 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:45.427024 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:45.427608 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:06:45.427813 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:45.427818 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:45.428581 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:45.428582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:45.429585 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:45.429587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:45.430005 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:45.430006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:45.430128 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:45.430718 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:06:45.430727 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:06:45.432203 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:06:45.434122 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:45.434130 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:45.434132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:45.434489 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:45.434919 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:45.434920 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:45.434920 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:45.435158 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:45.435440 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:06:45.435448 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:06:45.435603 (+ 155us) client_negotiation.cc:769] Sending connection context
0620 00:06:45.435711 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 00:06:45.435902 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":89,"threads_started":1}
I20260620 00:06:45.437332 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33831
I20260620 00:06:45.437476 1421 heartbeater.cc:464] Registering TS with master...
I20260620 00:06:45.437682 1421 heartbeater.cc:511] Master 127.0.98.62:33831 requested a full tablet report, sending...
I20260620 00:06:45.438203 962 ts_manager.cc:194] Registered new tserver with Master: 5006d1a946bc4ff2baca9653516b6b31 (127.0.98.3:39795)
I20260620 00:06:45.438602 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45179
I20260620 00:06:45.447992 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:06:45.456354 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:45.449718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57506 (local address 127.0.98.62:33831)
0620 00:06:45.449798 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:45.449800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:45.449811 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:06:45.450086 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:45.450088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:45.450098 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:45.450160 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:45.450766 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:45.451491 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:45.452215 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:45.452403 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:45.452463 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:45.453794 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:45.453795 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:45.453795 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:45.453802 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:45.455188 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:45.455584 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:45.455585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:45.455585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:45.455648 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:45.455878 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:45.455878 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:45.455879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:45.456031 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:45.456139 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:45.456216 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 00:06:45.456285 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 00:06:45.458859 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57506:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:06:45.460511 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:06:45.467691 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:45.463173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35888 (local address 127.0.98.3:39795)
0620 00:06:45.464007 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:45.464011 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:45.464022 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:06:45.464060 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:45.464063 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:45.464079 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:45.464151 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:06:45.464336 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:45.465043 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:45.466597 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:45.467202 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:45.467308 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:45.467377 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:45.467543 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 00:06:45.467584 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":791,"thread_start_us":75,"threads_started":1}
I20260620 00:06:45.467715 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:45.463090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39795 (local address 127.0.0.1:35888)
0620 00:06:45.463277 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:45.463318 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:45.463387 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:45.464228 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:45.464229 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:45.464230 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:06:45.464275 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:45.464277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:45.465157 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:45.465158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:45.466501 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:45.466504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:45.467311 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:45.467312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:45.467397 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:45.467413 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 00:06:45.467525 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 00:06:45.467592 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":114,"thread_start_us":53,"threads_started":1}
I20260620 00:06:45.467725 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:45.463101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46227 (local address 127.0.0.1:43732)
0620 00:06:45.463334 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:45.463346 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:45.463406 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:45.463703 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:45.463706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:45.463709 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:06:45.463790 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:45.463793 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:45.465221 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:45.465222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:45.466434 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:45.466437 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:45.467420 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:45.467422 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:45.467496 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:45.467501 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:06:45.467556 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 00:06:45.467608 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":33,"thread_start_us":93,"threads_started":1}
I20260620 00:06:45.467933 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:45.463336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43732 (local address 127.0.98.2:46227)
0620 00:06:45.463533 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:45.463540 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:45.463554 (+ 14us) server_negotiation.cc:407] Connection header received
0620 00:06:45.463587 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:45.463589 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:45.463602 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:45.463697 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:06:45.463892 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:45.465102 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:45.466522 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:45.467242 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:45.467325 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:45.467382 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:45.467768 (+ 386us) server_negotiation.cc:299] Negotiation successful
0620 00:06:45.467822 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":86,"threads_started":1}
I20260620 00:06:45.468449 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:45.463141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40705 (local address 127.0.0.1:37910)
0620 00:06:45.463793 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:45.463805 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:45.463857 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:45.464146 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:45.464147 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:45.464150 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:06:45.464199 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:45.464201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:45.465434 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:45.465435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:45.466693 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:45.466695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:45.467684 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:45.467685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:45.467753 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:45.467756 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:06:45.468316 (+ 560us) client_negotiation.cc:240] Negotiation successful
0620 00:06:45.468371 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":87,"threads_started":1}
I20260620 00:06:45.468564 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:45.463374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37910 (local address 127.0.98.1:40705)
0620 00:06:45.463561 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:45.463565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:45.463874 (+ 309us) server_negotiation.cc:407] Connection header received
0620 00:06:45.463925 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:45.463928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:45.463944 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:45.464064 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:06:45.464281 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:45.465311 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:45.466781 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:45.467536 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:45.467599 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:45.467649 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:45.468337 (+ 688us) server_negotiation.cc:299] Negotiation successful
0620 00:06:45.468470 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":74,"threads_started":1}
I20260620 00:06:45.468626 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 6f86307d36584c868805bd26da62c2ae (DEFAULT_TABLE table=test-table [id=d02f83988dc340e6a36b40d7ad305147]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:06:45.468855 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 6f86307d36584c868805bd26da62c2ae (DEFAULT_TABLE table=test-table [id=d02f83988dc340e6a36b40d7ad305147]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:06:45.469089 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f86307d36584c868805bd26da62c2ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:06:45.469215 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f86307d36584c868805bd26da62c2ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:06:45.469496 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 6f86307d36584c868805bd26da62c2ae (DEFAULT_TABLE table=test-table [id=d02f83988dc340e6a36b40d7ad305147]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:06:45.469811 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f86307d36584c868805bd26da62c2ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:06:45.472165 1438 tablet_bootstrap.cc:492] T 6f86307d36584c868805bd26da62c2ae P 565c0bec474d4984801748f4d3441570: Bootstrap starting.
I20260620 00:06:45.472620 1439 tablet_bootstrap.cc:492] T 6f86307d36584c868805bd26da62c2ae P 5006d1a946bc4ff2baca9653516b6b31: Bootstrap starting.
I20260620 00:06:45.473163 1440 tablet_bootstrap.cc:492] T 6f86307d36584c868805bd26da62c2ae P ea54912757f743de96d4629a7f97f37c: Bootstrap starting.
I20260620 00:06:45.473791 1440 tablet_bootstrap.cc:654] T 6f86307d36584c868805bd26da62c2ae P ea54912757f743de96d4629a7f97f37c: Neither blocks nor log segments found. Creating new log.
I20260620 00:06:45.473793 1438 tablet_bootstrap.cc:654] T 6f86307d36584c868805bd26da62c2ae P 565c0bec474d4984801748f4d3441570: Neither blocks nor log segments found. Creating new log.
I20260620 00:06:45.473793 1439 tablet_bootstrap.cc:654] T 6f86307d36584c868805bd26da62c2ae P 5006d1a946bc4ff2baca9653516b6b31: Neither blocks nor log segments found. Creating new log.
I20260620 00:06:45.474287 1440 log.cc:824] T 6f86307d36584c868805bd26da62c2ae P ea54912757f743de96d4629a7f97f37c: Log is configured to *not* fsync() on all Append() calls
I20260620 00:06:45.474409 1438 log.cc:824] T 6f86307d36584c868805bd26da62c2ae P 565c0bec474d4984801748f4d3441570: Log is configured to *not* fsync() on all Append() calls
I20260620 00:06:45.474409 1439 log.cc:824] T 6f86307d36584c868805bd26da62c2ae P 5006d1a946bc4ff2baca9653516b6b31: Log is configured to *not* fsync() on all Append() calls
I20260620 00:06:45.475207 1438 tablet_bootstrap.cc:492] T 6f86307d36584c868805bd26da62c2ae P 565c0bec474d4984801748f4d3441570: No bootstrap required, opened a new log
I20260620 00:06:45.475207 1440 tablet_bootstrap.cc:492] T 6f86307d36584c868805bd26da62c2ae P ea54912757f743de96d4629a7f97f37c: No bootstrap required, opened a new log
I20260620 00:06:45.475286 1440 ts_tablet_manager.cc:1397] T 6f86307d36584c868805bd26da62c2ae P ea54912757f743de96d4629a7f97f37c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:06:45.475286 1438 ts_tablet_manager.cc:1397] T 6f86307d36584c868805bd26da62c2ae P 565c0bec474d4984801748f4d3441570: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 00:06:45.475355 1439 tablet_bootstrap.cc:492] T 6f86307d36584c868805bd26da62c2ae P 5006d1a946bc4ff2baca9653516b6b31: No bootstrap required, opened a new log
I20260620 00:06:45.475426 1439 ts_tablet_manager.cc:1397] T 6f86307d36584c868805bd26da62c2ae P 5006d1a946bc4ff2baca9653516b6b31: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 00:06:45.477113 1439 raft_consensus.cc:348] T 6f86307d36584c868805bd26da62c2ae P 5006d1a946bc4ff2baca9653516b6b31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5006d1a946bc4ff2baca9653516b6b31" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39795 } } peers { permanent_uuid: "565c0bec474d4984801748f4d3441570" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46227 } } peers { permanent_uuid: "ea54912757f743de96d4629a7f97f37c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40705 } }
I20260620 00:06:45.477303 1439 raft_consensus.cc:374] T 6f86307d36584c868805bd26da62c2ae P 5006d1a946bc4ff2baca9653516b6b31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:06:45.477347 1439 raft_consensus.cc:729] T 6f86307d36584c868805bd26da62c2ae P 5006d1a946bc4ff2baca9653516b6b31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5006d1a946bc4ff2baca9653516b6b31, State: Initialized, Role: FOLLOWER
I20260620 00:06:45.477540 1439 consensus_queue.cc:260] T 6f86307d36584c868805bd26da62c2ae P 5006d1a946bc4ff2baca9653516b6b31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5006d1a946bc4ff2baca9653516b6b31" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39795 } } peers { permanent_uuid: "565c0bec474d4984801748f4d3441570" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46227 } } peers { permanent_uuid: "ea54912757f743de96d4629a7f97f37c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40705 } }
I20260620 00:06:45.477423 1438 raft_consensus.cc:348] T 6f86307d36584c868805bd26da62c2ae P 565c0bec474d4984801748f4d3441570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5006d1a946bc4ff2baca9653516b6b31" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39795 } } peers { permanent_uuid: "565c0bec474d4984801748f4d3441570" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46227 } } peers { permanent_uuid: "ea54912757f743de96d4629a7f97f37c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40705 } }
I20260620 00:06:45.477423 1440 raft_consensus.cc:348] T 6f86307d36584c868805bd26da62c2ae P ea54912757f743de96d4629a7f97f37c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5006d1a946bc4ff2baca9653516b6b31" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39795 } } peers { permanent_uuid: "565c0bec474d4984801748f4d3441570" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46227 } } peers { permanent_uuid: "ea54912757f743de96d4629a7f97f37c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40705 } }
I20260620 00:06:45.477638 1438 raft_consensus.cc:374] T 6f86307d36584c868805bd26da62c2ae P 565c0bec474d4984801748f4d3441570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:06:45.477638 1440 raft_consensus.cc:374] T 6f86307d36584c868805bd26da62c2ae P ea54912757f743de96d4629a7f97f37c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:06:45.477684 1438 raft_consensus.cc:729] T 6f86307d36584c868805bd26da62c2ae P 565c0bec474d4984801748f4d3441570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 565c0bec474d4984801748f4d3441570, State: Initialized, Role: FOLLOWER
I20260620 00:06:45.477684 1440 raft_consensus.cc:729] T 6f86307d36584c868805bd26da62c2ae P ea54912757f743de96d4629a7f97f37c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea54912757f743de96d4629a7f97f37c, State: Initialized, Role: FOLLOWER
I20260620 00:06:45.477810 1438 consensus_queue.cc:260] T 6f86307d36584c868805bd26da62c2ae P 565c0bec474d4984801748f4d3441570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5006d1a946bc4ff2baca9653516b6b31" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39795 } } peers { permanent_uuid: "565c0bec474d4984801748f4d3441570" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46227 } } peers { permanent_uuid: "ea54912757f743de96d4629a7f97f37c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40705 } }
I20260620 00:06:45.477803 1440 consensus_queue.cc:260] T 6f86307d36584c868805bd26da62c2ae P ea54912757f743de96d4629a7f97f37c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5006d1a946bc4ff2baca9653516b6b31" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39795 } } peers { permanent_uuid: "565c0bec474d4984801748f4d3441570" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46227 } } peers { permanent_uuid: "ea54912757f743de96d4629a7f97f37c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40705 } }
I20260620 00:06:45.478097 1440 ts_tablet_manager.cc:1428] T 6f86307d36584c868805bd26da62c2ae P ea54912757f743de96d4629a7f97f37c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:06:45.478097 1438 ts_tablet_manager.cc:1428] T 6f86307d36584c868805bd26da62c2ae P 565c0bec474d4984801748f4d3441570: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260620 00:06:45.478221 1287 heartbeater.cc:503] Master 127.0.98.62:33831 was elected leader, sending a full tablet report...
I20260620 00:06:45.478355 1153 heartbeater.cc:503] Master 127.0.98.62:33831 was elected leader, sending a full tablet report...
I20260620 00:06:45.478350 1439 ts_tablet_manager.cc:1428] T 6f86307d36584c868805bd26da62c2ae P 5006d1a946bc4ff2baca9653516b6b31: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 00:06:45.478467 1421 heartbeater.cc:503] Master 127.0.98.62:33831 was elected leader, sending a full tablet report...
W20260620 00:06:45.520037 1288 tablet.cc:2369] T 6f86307d36584c868805bd26da62c2ae P 565c0bec474d4984801748f4d3441570: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:06:45.532953 1445 raft_consensus.cc:482] T 6f86307d36584c868805bd26da62c2ae P 565c0bec474d4984801748f4d3441570 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:06:45.533177 1445 raft_consensus.cc:504] T 6f86307d36584c868805bd26da62c2ae P 565c0bec474d4984801748f4d3441570 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5006d1a946bc4ff2baca9653516b6b31" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39795 } } peers { permanent_uuid: "565c0bec474d4984801748f4d3441570" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46227 } } peers { permanent_uuid: "ea54912757f743de96d4629a7f97f37c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40705 } }
I20260620 00:06:45.533659 1445 leader_election.cc:288] T 6f86307d36584c868805bd26da62c2ae P 565c0bec474d4984801748f4d3441570 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5006d1a946bc4ff2baca9653516b6b31 (127.0.98.3:39795), ea54912757f743de96d4629a7f97f37c (127.0.98.1:40705)
I20260620 00:06:45.538129 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:45.534043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46525 (local address 127.0.98.3:39795)
0620 00:06:45.534097 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:45.534099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:45.534115 (+ 16us) server_negotiation.cc:407] Connection header received
0620 00:06:45.534272 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:45.534274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:45.534289 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:45.534362 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:06:45.534668 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:45.535422 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:45.537052 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:45.537693 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:45.537777 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:45.537790 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:45.538003 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 00:06:45.538047 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 00:06:45.538129 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:45.534015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35057 (local address 127.0.98.1:40705)
0620 00:06:45.534072 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:45.534074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:45.534239 (+ 165us) server_negotiation.cc:407] Connection header received
0620 00:06:45.534277 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:45.534277 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:45.534289 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:45.534362 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:06:45.534668 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:45.535438 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:45.537077 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:45.537695 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:45.537778 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:45.537790 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:45.538003 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 00:06:45.538059 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 00:06:45.538162 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:45.533928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39795 (local address 127.0.98.2:46525)
0620 00:06:45.534043 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:45.534059 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:45.534132 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:45.534413 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:45.534414 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:45.534417 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:06:45.534527 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:45.534530 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:45.535538 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:45.535539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:45.536931 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:45.536933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:45.537811 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:45.537812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:45.537910 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:45.537913 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:06:45.537986 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 00:06:45.538044 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":82}
I20260620 00:06:45.538170 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:45.533928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40705 (local address 127.0.98.2:35057)
0620 00:06:45.534144 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:45.534153 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:45.534211 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:45.534413 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:45.534414 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:45.534417 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:06:45.534562 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:45.534564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:45.535544 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:45.535545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:45.536927 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:45.536930 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:45.537811 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:45.537812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:45.537908 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:45.537913 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:06:45.537985 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 00:06:45.538044 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":131,"thread_start_us":113,"threads_started":1}
I20260620 00:06:45.538599 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f86307d36584c868805bd26da62c2ae" candidate_uuid: "565c0bec474d4984801748f4d3441570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5006d1a946bc4ff2baca9653516b6b31" is_pre_election: true
I20260620 00:06:45.538797 1375 raft_consensus.cc:2454] T 6f86307d36584c868805bd26da62c2ae P 5006d1a946bc4ff2baca9653516b6b31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 565c0bec474d4984801748f4d3441570 in term 0.
I20260620 00:06:45.538931 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f86307d36584c868805bd26da62c2ae" candidate_uuid: "565c0bec474d4984801748f4d3441570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea54912757f743de96d4629a7f97f37c" is_pre_election: true
I20260620 00:06:45.539113 1106 raft_consensus.cc:2454] T 6f86307d36584c868805bd26da62c2ae P ea54912757f743de96d4629a7f97f37c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 565c0bec474d4984801748f4d3441570 in term 0.
I20260620 00:06:45.539382 1174 leader_election.cc:302] T 6f86307d36584c868805bd26da62c2ae P 565c0bec474d4984801748f4d3441570 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 565c0bec474d4984801748f4d3441570, ea54912757f743de96d4629a7f97f37c; no voters:
I20260620 00:06:45.539564 1445 raft_consensus.cc:2781] T 6f86307d36584c868805bd26da62c2ae P 565c0bec474d4984801748f4d3441570 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:06:45.539656 1445 raft_consensus.cc:482] T 6f86307d36584c868805bd26da62c2ae P 565c0bec474d4984801748f4d3441570 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:06:45.539674 1445 raft_consensus.cc:3037] T 6f86307d36584c868805bd26da62c2ae P 565c0bec474d4984801748f4d3441570 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:06:45.540508 1445 raft_consensus.cc:504] T 6f86307d36584c868805bd26da62c2ae P 565c0bec474d4984801748f4d3441570 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5006d1a946bc4ff2baca9653516b6b31" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39795 } } peers { permanent_uuid: "565c0bec474d4984801748f4d3441570" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46227 } } peers { permanent_uuid: "ea54912757f743de96d4629a7f97f37c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40705 } }
I20260620 00:06:45.540740 1445 leader_election.cc:288] T 6f86307d36584c868805bd26da62c2ae P 565c0bec474d4984801748f4d3441570 [CANDIDATE]: Term 1 election: Requested vote from peers 5006d1a946bc4ff2baca9653516b6b31 (127.0.98.3:39795), ea54912757f743de96d4629a7f97f37c (127.0.98.1:40705)
I20260620 00:06:45.540937 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f86307d36584c868805bd26da62c2ae" candidate_uuid: "565c0bec474d4984801748f4d3441570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5006d1a946bc4ff2baca9653516b6b31"
I20260620 00:06:45.540967 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f86307d36584c868805bd26da62c2ae" candidate_uuid: "565c0bec474d4984801748f4d3441570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea54912757f743de96d4629a7f97f37c"
I20260620 00:06:45.541028 1375 raft_consensus.cc:3037] T 6f86307d36584c868805bd26da62c2ae P 5006d1a946bc4ff2baca9653516b6b31 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:06:45.541098 1106 raft_consensus.cc:3037] T 6f86307d36584c868805bd26da62c2ae P ea54912757f743de96d4629a7f97f37c [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:06:45.541952 1375 raft_consensus.cc:2454] T 6f86307d36584c868805bd26da62c2ae P 5006d1a946bc4ff2baca9653516b6b31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 565c0bec474d4984801748f4d3441570 in term 1.
I20260620 00:06:45.541968 1106 raft_consensus.cc:2454] T 6f86307d36584c868805bd26da62c2ae P ea54912757f743de96d4629a7f97f37c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 565c0bec474d4984801748f4d3441570 in term 1.
I20260620 00:06:45.542183 1176 leader_election.cc:302] T 6f86307d36584c868805bd26da62c2ae P 565c0bec474d4984801748f4d3441570 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5006d1a946bc4ff2baca9653516b6b31, 565c0bec474d4984801748f4d3441570; no voters:
I20260620 00:06:45.542297 1445 raft_consensus.cc:2781] T 6f86307d36584c868805bd26da62c2ae P 565c0bec474d4984801748f4d3441570 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:06:45.542459 1445 raft_consensus.cc:686] T 6f86307d36584c868805bd26da62c2ae P 565c0bec474d4984801748f4d3441570 [term 1 LEADER]: Becoming Leader. State: Replica: 565c0bec474d4984801748f4d3441570, State: Running, Role: LEADER
I20260620 00:06:45.542577 1445 consensus_queue.cc:237] T 6f86307d36584c868805bd26da62c2ae P 565c0bec474d4984801748f4d3441570 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5006d1a946bc4ff2baca9653516b6b31" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39795 } } peers { permanent_uuid: "565c0bec474d4984801748f4d3441570" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46227 } } peers { permanent_uuid: "ea54912757f743de96d4629a7f97f37c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40705 } }
I20260620 00:06:45.543602 962 catalog_manager.cc:5495] T 6f86307d36584c868805bd26da62c2ae P 565c0bec474d4984801748f4d3441570 reported cstate change: term changed from 0 to 1, leader changed from <none> to 565c0bec474d4984801748f4d3441570 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "565c0bec474d4984801748f4d3441570" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5006d1a946bc4ff2baca9653516b6b31" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "565c0bec474d4984801748f4d3441570" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea54912757f743de96d4629a7f97f37c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40705 } health_report { overall_health: UNKNOWN } } }
Jun 20 00:06:45 dist-test-slave-grqt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 00:06:45.567409 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:45.559196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37922 (local address 127.0.98.1:40705)
0620 00:06:45.559251 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:45.559253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:45.559339 (+ 86us) server_negotiation.cc:407] Connection header received
0620 00:06:45.559381 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:45.559382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:45.559394 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:45.559448 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:45.560132 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:45.560610 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:45.561425 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:45.561632 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:45.561709 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:45.564243 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:45.564256 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:45.564256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:45.564264 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:45.566116 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:45.566534 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:45.566539 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:45.566539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:45.566619 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:45.566815 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:45.566816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:45.566816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:45.566979 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:45.567145 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:45.567212 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 00:06:45.567287 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Jun 20 00:06:45 dist-test-slave-grqt krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914005, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 00:06:45 dist-test-slave-grqt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914005, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 00:06:45.593990 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:45.585417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37932 (local address 127.0.98.1:40705)
0620 00:06:45.585468 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:45.585470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:45.585650 (+ 180us) server_negotiation.cc:407] Connection header received
0620 00:06:45.585737 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:45.585738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:45.585800 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:45.585912 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:45.586597 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:45.587095 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:45.587893 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:45.588211 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:45.588291 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:45.591031 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:45.591032 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:45.591033 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:45.591042 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:45.592576 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:45.593002 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:45.593003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:45.593003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:45.593080 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:45.593284 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:45.593285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:45.593286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:45.593491 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:45.593600 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:45.593787 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 00:06:45.593880 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 00:06:45.605221 1154 tablet.cc:2369] T 6f86307d36584c868805bd26da62c2ae P ea54912757f743de96d4629a7f97f37c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 00:06:45 dist-test-slave-grqt krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 00:06:45 dist-test-slave-grqt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 00:06:45.620061 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:45.611872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37940 (local address 127.0.98.1:40705)
0620 00:06:45.611943 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:45.611945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:45.612006 (+ 61us) server_negotiation.cc:407] Connection header received
0620 00:06:45.612074 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:45.612075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:45.612085 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:45.612161 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:45.612869 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:45.613490 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:45.614365 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:45.614571 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:45.614638 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:45.617284 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:45.617286 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:45.617286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:45.617294 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:45.618853 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:45.619225 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:45.619226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:45.619226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:45.619280 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:45.619570 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:45.619571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:45.619571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:45.619674 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:45.619749 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:45.619920 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 00:06:45.619990 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 00:06:45.620594 1067 tablet_service.cc:2908] Rejecting scan request for tablet 6f86307d36584c868805bd26da62c2ae: Uninitialized: clean time has not yet been initialized
I20260620 00:06:45.621634 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 1024
I20260620 00:06:45.627198 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 1158
I20260620 00:06:45.631949 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 1292
I20260620 00:06:45.636735 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 930
2026-06-20T00:06:45Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2029 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:06:45 dist-test-slave-grqt krb5kdc[1462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:06:45 dist-test-slave-grqt krb5kdc[1462](info): set up 2 sockets
Jun 20 00:06:45 dist-test-slave-grqt krb5kdc[1462](info): commencing operation
krb5kdc: starting...
W20260620 00:06:46.689512 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:06:46 dist-test-slave-grqt krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:06:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:06:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:06:46.860879 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39169
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:06:46.950388 1475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:06:46.950611 1475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:06:46.950635 1475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:06:46.951897 1475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:06:46.951944 1475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 00:06:46.952008 1475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:06:46.952040 1475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:06:46.952059 1475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:06:46.952090 1475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:06:46.953747 1475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39169
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:06:46.954028 1475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:06:46.954283 1475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:06:46.959355 1481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:06:46.959349 1483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:06:46.959429 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:06:46.959723 1475 server_base.cc:1034] running on GCE node
I20260620 00:06:46.960005 1475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:06:46.960330 1475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:06:46.961484 1475 hybrid_clock.cc:648] HybridClock initialized: now 1781914006961475 us; error 36 us; skew 500 ppm
Jun 20 00:06:46 dist-test-slave-grqt krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:06:46.964190 1475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:06:46.964706 1475 webserver.cc:460] Webserver started at http://127.0.98.62:46557/ using document root <none> and password file <none>
I20260620 00:06:46.964953 1475 fs_manager.cc:359] Metadata directory not provided
I20260620 00:06:46.965024 1475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:06:46.965173 1475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:06:46.966326 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "3fed10f0e8e94861bc50af06634490e1"
format_stamp: "Formatted at 2026-06-20 00:06:46 on dist-test-slave-grqt"
server_key: "bd511a064cf0f92daab902e7259ae665"
server_key_iv: "b6650375e77e6d23fc826152bad644f9"
server_key_version: "encryptionkey@0"
I20260620 00:06:46.966696 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "3fed10f0e8e94861bc50af06634490e1"
format_stamp: "Formatted at 2026-06-20 00:06:46 on dist-test-slave-grqt"
server_key: "bd511a064cf0f92daab902e7259ae665"
server_key_iv: "b6650375e77e6d23fc826152bad644f9"
server_key_version: "encryptionkey@0"
I20260620 00:06:46.968174 1475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:06:46.969060 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:46.969250 1475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:06:46.969357 1475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "3fed10f0e8e94861bc50af06634490e1"
format_stamp: "Formatted at 2026-06-20 00:06:46 on dist-test-slave-grqt"
server_key: "bd511a064cf0f92daab902e7259ae665"
server_key_iv: "b6650375e77e6d23fc826152bad644f9"
server_key_version: "encryptionkey@0"
I20260620 00:06:46.969463 1475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:06:47.008513 1475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:06:47.011108 1475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:06:47.011294 1475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:06:47.016266 1475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39169
I20260620 00:06:47.016297 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39169 every 8 connection(s)
I20260620 00:06:47.016667 1475 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:06:47.017210 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 1475
I20260620 00:06:47.017287 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:06:47.017491 392 external_mini_cluster.cc:1428] Setting key 977b302c66dad307809328cd0fb0cc4f
I20260620 00:06:47.017598 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:06:47.021061 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fed10f0e8e94861bc50af06634490e1: Bootstrap starting.
I20260620 00:06:47.021865 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fed10f0e8e94861bc50af06634490e1: Neither blocks nor log segments found. Creating new log.
I20260620 00:06:47.022171 1553 log.cc:824] T 00000000000000000000000000000000 P 3fed10f0e8e94861bc50af06634490e1: Log is configured to *not* fsync() on all Append() calls
I20260620 00:06:47.023149 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fed10f0e8e94861bc50af06634490e1: No bootstrap required, opened a new log
Jun 20 00:06:47 dist-test-slave-grqt krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:06:47.025557 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fed10f0e8e94861bc50af06634490e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fed10f0e8e94861bc50af06634490e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39169 } }
I20260620 00:06:47.025763 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fed10f0e8e94861bc50af06634490e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:06:47.025810 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fed10f0e8e94861bc50af06634490e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fed10f0e8e94861bc50af06634490e1, State: Initialized, Role: FOLLOWER
I20260620 00:06:47.025990 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fed10f0e8e94861bc50af06634490e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fed10f0e8e94861bc50af06634490e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39169 } }
I20260620 00:06:47.026072 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fed10f0e8e94861bc50af06634490e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:06:47.026096 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fed10f0e8e94861bc50af06634490e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:06:47.026166 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fed10f0e8e94861bc50af06634490e1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:06:47.026834 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fed10f0e8e94861bc50af06634490e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fed10f0e8e94861bc50af06634490e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39169 } }
I20260620 00:06:47.026924 1553 leader_election.cc:302] T 00000000000000000000000000000000 P 3fed10f0e8e94861bc50af06634490e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fed10f0e8e94861bc50af06634490e1; no voters:
I20260620 00:06:47.027144 1553 leader_election.cc:288] T 00000000000000000000000000000000 P 3fed10f0e8e94861bc50af06634490e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:06:47.027174 1558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fed10f0e8e94861bc50af06634490e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:06:47.027283 1558 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fed10f0e8e94861bc50af06634490e1 [term 1 LEADER]: Becoming Leader. State: Replica: 3fed10f0e8e94861bc50af06634490e1, State: Running, Role: LEADER
I20260620 00:06:47.027366 1558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fed10f0e8e94861bc50af06634490e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fed10f0e8e94861bc50af06634490e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39169 } }
I20260620 00:06:47.027438 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fed10f0e8e94861bc50af06634490e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:06:47.027714 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fed10f0e8e94861bc50af06634490e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fed10f0e8e94861bc50af06634490e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fed10f0e8e94861bc50af06634490e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39169 } } }
I20260620 00:06:47.027993 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fed10f0e8e94861bc50af06634490e1 [sys.catalog]: This master's current role is: LEADER
I20260620 00:06:47.027889 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fed10f0e8e94861bc50af06634490e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fed10f0e8e94861bc50af06634490e1. Latest consensus state: current_term: 1 leader_uuid: "3fed10f0e8e94861bc50af06634490e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fed10f0e8e94861bc50af06634490e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39169 } } }
I20260620 00:06:47.028272 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fed10f0e8e94861bc50af06634490e1 [sys.catalog]: This master's current role is: LEADER
I20260620 00:06:47.028443 1567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:06:47.028970 1567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:06:47.029441 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:47.018050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48774 (local address 127.0.98.62:39169)
0620 00:06:47.018246 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:47.018255 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:47.018292 (+ 37us) server_negotiation.cc:407] Connection header received
0620 00:06:47.018752 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:47.018758 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:47.018811 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:47.018931 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:47.019764 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.020703 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.021763 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.022278 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.022382 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:47.025180 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:47.025184 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:47.025185 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:47.025207 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:47.027456 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:47.027873 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:47.027876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:47.027876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:47.028031 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:47.028266 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:47.028267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:47.028268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:47.028621 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:47.028775 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:47.029049 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 00:06:47.029213 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":104,"threads_started":1}
I20260620 00:06:47.030998 1567 catalog_manager.cc:1347] Generated new cluster ID: 59edb84a133b4b7ea6011a35a48e7e59
I20260620 00:06:47.031049 1567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:06:47.051959 1567 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:06:47.052544 1567 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:06:47.056073 1567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fed10f0e8e94861bc50af06634490e1: Generated new TSK 0
I20260620 00:06:47.056268 1567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:06:47.084559 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39169
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:06:47.167845 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:06:47.168066 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:06:47.168093 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:06:47.169353 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:06:47.169446 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:06:47.169471 1579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 00:06:47.169508 1579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:06:47.171092 1579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:06:47.171347 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:06:47.171600 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:06:47.177210 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:06:47.177219 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:06:47.177273 1579 server_base.cc:1034] running on GCE node
W20260620 00:06:47.177219 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:06:47.177698 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:06:47.178000 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:06:47.179193 1579 hybrid_clock.cc:648] HybridClock initialized: now 1781914007179182 us; error 48 us; skew 500 ppm
Jun 20 00:06:47 dist-test-slave-grqt krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:06:47.181979 1579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:06:47.182489 1579 webserver.cc:460] Webserver started at http://127.0.98.1:45857/ using document root <none> and password file <none>
I20260620 00:06:47.182726 1579 fs_manager.cc:359] Metadata directory not provided
I20260620 00:06:47.182796 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:06:47.182948 1579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:06:47.184058 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-0/data/instance:
uuid: "6692ef81cd7a458aaa76cf9c8d53258d"
format_stamp: "Formatted at 2026-06-20 00:06:47 on dist-test-slave-grqt"
server_key: "43b22d62bffbca43ecb6fb7f4b96eff7"
server_key_iv: "a2b76f61c4f95d0881fdd5014fb81eda"
server_key_version: "encryptionkey@0"
I20260620 00:06:47.184448 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6692ef81cd7a458aaa76cf9c8d53258d"
format_stamp: "Formatted at 2026-06-20 00:06:47 on dist-test-slave-grqt"
server_key: "43b22d62bffbca43ecb6fb7f4b96eff7"
server_key_iv: "a2b76f61c4f95d0881fdd5014fb81eda"
server_key_version: "encryptionkey@0"
I20260620 00:06:47.185930 1579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:06:47.186793 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:47.187007 1579 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:06:47.187114 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-0/wal
uuid: "6692ef81cd7a458aaa76cf9c8d53258d"
format_stamp: "Formatted at 2026-06-20 00:06:47 on dist-test-slave-grqt"
server_key: "43b22d62bffbca43ecb6fb7f4b96eff7"
server_key_iv: "a2b76f61c4f95d0881fdd5014fb81eda"
server_key_version: "encryptionkey@0"
I20260620 00:06:47.187211 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:06:47.198686 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:06:47.200925 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:06:47.201099 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:06:47.201817 1579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:06:47.201870 1579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:47.201905 1579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:06:47.201924 1579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 00:06:47 dist-test-slave-grqt krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:06:47.209333 1579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35335
I20260620 00:06:47.209774 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:06:47.209790 1709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35335 every 8 connection(s)
I20260620 00:06:47.211849 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:47.202247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40975 (local address 127.0.98.62:39169)
0620 00:06:47.202311 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:47.202314 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:47.202578 (+ 264us) server_negotiation.cc:407] Connection header received
0620 00:06:47.203343 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:47.203344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:47.203355 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:47.203400 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:47.204390 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.204969 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.206223 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.206535 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.206639 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:47.208882 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:47.208883 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:47.208884 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:47.208891 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:47.210407 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:47.210880 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:47.210882 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:47.210883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:47.210944 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:47.211205 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:47.211206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:47.211206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:47.211320 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:47.211392 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:47.211668 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 00:06:47.211756 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 00:06:47.212047 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:47.202245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39169 (local address 127.0.98.1:40975)
0620 00:06:47.202409 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:47.202467 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:47.203222 (+ 755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:47.203486 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:47.203490 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:47.203912 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:06:47.204258 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:47.204265 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.205103 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.205104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:47.206136 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:47.206139 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.206667 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.206669 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:47.206787 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:47.207364 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:06:47.207372 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:06:47.208778 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:06:47.210501 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:47.210506 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:47.210508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:47.210788 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:47.211018 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:47.211019 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:47.211019 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:47.211131 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:47.211384 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:06:47.211389 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:06:47.211530 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 00:06:47.211649 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 00:06:47.211811 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":81,"threads_started":1}
I20260620 00:06:47.212545 1710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39169
I20260620 00:06:47.212652 1710 heartbeater.cc:464] Registering TS with master...
I20260620 00:06:47.212834 1710 heartbeater.cc:511] Master 127.0.98.62:39169 requested a full tablet report, sending...
I20260620 00:06:47.213476 1506 ts_manager.cc:194] Registered new tserver with Master: 6692ef81cd7a458aaa76cf9c8d53258d (127.0.98.1:35335)
I20260620 00:06:47.214422 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40975
I20260620 00:06:47.219168 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:47.213689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39475 (local address 127.0.98.62:39169)
0620 00:06:47.213729 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:47.213730 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:47.213890 (+ 160us) server_negotiation.cc:407] Connection header received
0620 00:06:47.213913 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:47.213913 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:47.213922 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:47.213968 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:47.214314 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.214850 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.215635 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.215833 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.215904 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:47.216814 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:47.216816 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:47.216816 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:47.216822 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:47.218018 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:47.218388 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:47.218389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:47.218390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:47.218444 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:47.218642 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:47.218643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:47.218643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:47.218749 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:47.218823 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:47.218972 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 00:06:47.219051 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 00:06:47.219170 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:47.213641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39169 (local address 127.0.98.1:39475)
0620 00:06:47.213791 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:47.213803 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:47.213868 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:47.213972 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:47.213973 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:47.214182 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:06:47.214249 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:47.214251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.214943 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.214944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:47.215525 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:47.215527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.215913 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.215914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:47.215979 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:47.216401 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:06:47.216405 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:06:47.216710 (+ 305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:06:47.218120 (+ 1410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:47.218121 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:47.218122 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:47.218302 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:47.218507 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:47.218508 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:47.218509 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:47.218566 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:47.218813 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:06:47.218814 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:06:47.218858 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 00:06:47.218948 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 00:06:47.219057 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":27,"thread_start_us":66,"threads_started":1}
I20260620 00:06:47.220027 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 1579
I20260620 00:06:47.220111 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:06:47.220297 392 external_mini_cluster.cc:1428] Setting key 6998074895d1e069c69cd15561bcc5dd
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:06:47.244226 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39169
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:06:47.325300 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:06:47.325539 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:06:47.325564 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:06:47.326783 1719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:06:47.326831 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:06:47.326853 1719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 00:06:47.326870 1719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:06:47.328619 1719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:06:47.328861 1719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:06:47.329121 1719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:06:47.334167 1724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:06:47.334275 1719 server_base.cc:1034] running on GCE node
W20260620 00:06:47.334241 1727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:06:47.334165 1725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:06:47.334584 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:06:47.334787 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:06:47.335939 1719 hybrid_clock.cc:648] HybridClock initialized: now 1781914007335915 us; error 42 us; skew 500 ppm
Jun 20 00:06:47 dist-test-slave-grqt krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:06:47.338474 1719 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:06:47.338953 1719 webserver.cc:460] Webserver started at http://127.0.98.2:42475/ using document root <none> and password file <none>
I20260620 00:06:47.339175 1719 fs_manager.cc:359] Metadata directory not provided
I20260620 00:06:47.339250 1719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:06:47.339399 1719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:06:47.340502 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-1/data/instance:
uuid: "ce6fa384eddd46928bccb5c5d7b5650d"
format_stamp: "Formatted at 2026-06-20 00:06:47 on dist-test-slave-grqt"
server_key: "5cb3a7a7462d7e265ee024d0ab035400"
server_key_iv: "a3ebad196cecda71b72c825ff118200c"
server_key_version: "encryptionkey@0"
I20260620 00:06:47.340867 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ce6fa384eddd46928bccb5c5d7b5650d"
format_stamp: "Formatted at 2026-06-20 00:06:47 on dist-test-slave-grqt"
server_key: "5cb3a7a7462d7e265ee024d0ab035400"
server_key_iv: "a3ebad196cecda71b72c825ff118200c"
server_key_version: "encryptionkey@0"
I20260620 00:06:47.342312 1719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 00:06:47.343048 1734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:47.343233 1719 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:06:47.343335 1719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-1/wal
uuid: "ce6fa384eddd46928bccb5c5d7b5650d"
format_stamp: "Formatted at 2026-06-20 00:06:47 on dist-test-slave-grqt"
server_key: "5cb3a7a7462d7e265ee024d0ab035400"
server_key_iv: "a3ebad196cecda71b72c825ff118200c"
server_key_version: "encryptionkey@0"
I20260620 00:06:47.343429 1719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:06:47.369208 1719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:06:47.371294 1719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:06:47.371483 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:06:47.372148 1719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:06:47.372211 1719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:47.372269 1719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:06:47.372298 1719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:47.378782 1719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43591
I20260620 00:06:47.378881 1849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43591 every 8 connection(s)
I20260620 00:06:47.379266 1719 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 00:06:47 dist-test-slave-grqt krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:06:47.382866 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:47.372684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51427 (local address 127.0.98.62:39169)
0620 00:06:47.372757 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:47.372760 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:47.372933 (+ 173us) server_negotiation.cc:407] Connection header received
0620 00:06:47.373628 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:47.373630 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:47.373645 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:47.373733 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:47.374650 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.375455 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.377079 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.377348 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.377457 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:47.380183 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:47.380185 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:47.380185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:47.380192 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:47.381708 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:47.382100 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:47.382101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:47.382101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:47.382150 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:47.382355 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:47.382356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:47.382356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:47.382455 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:47.382504 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:47.382737 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 00:06:47.382799 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 00:06:47.383107 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:47.372666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39169 (local address 127.0.98.2:51427)
0620 00:06:47.372815 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:47.372858 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:47.373520 (+ 662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:47.373822 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:47.373827 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:47.374237 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:06:47.374544 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:47.374549 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.375553 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.375554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:47.376975 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:47.376977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.377465 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.377466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:47.377614 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:47.378195 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:06:47.378207 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:06:47.380085 (+ 1878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:06:47.381790 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:47.381796 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:47.381798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:47.382019 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:47.382204 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:47.382204 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:47.382205 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:47.382301 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:47.382508 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:06:47.382511 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:06:47.382629 (+ 118us) client_negotiation.cc:769] Sending connection context
0620 00:06:47.382709 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 00:06:47.382879 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
I20260620 00:06:47.383639 1850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39169
I20260620 00:06:47.383714 1850 heartbeater.cc:464] Registering TS with master...
I20260620 00:06:47.383889 1850 heartbeater.cc:511] Master 127.0.98.62:39169 requested a full tablet report, sending...
I20260620 00:06:47.384271 1507 ts_manager.cc:194] Registered new tserver with Master: ce6fa384eddd46928bccb5c5d7b5650d (127.0.98.2:43591)
I20260620 00:06:47.384765 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51427
I20260620 00:06:47.389540 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 1719
I20260620 00:06:47.389638 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:06:47.389843 392 external_mini_cluster.cc:1428] Setting key 76998d8d6c07540c74ca0efa81297e2a
I20260620 00:06:47.390091 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:47.384722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48029 (local address 127.0.98.62:39169)
0620 00:06:47.384762 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:47.384763 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:47.384896 (+ 133us) server_negotiation.cc:407] Connection header received
0620 00:06:47.384973 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:47.384973 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:47.384983 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:47.385039 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:47.385417 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.385884 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.386535 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.386707 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.386758 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:47.387547 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:47.387548 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:47.387549 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:47.387554 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:47.388901 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:47.389187 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:47.389188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:47.389188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:47.389227 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:47.389404 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:47.389405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:47.389405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:47.389601 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:47.389678 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:47.389868 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 00:06:47.389943 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 00:06:47.390096 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:47.384659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39169 (local address 127.0.98.2:48029)
0620 00:06:47.384814 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:47.384831 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:47.384919 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:47.385047 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:47.385048 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:47.385255 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:06:47.385314 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:47.385316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.385968 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.385969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:47.386453 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:47.386454 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.386771 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.386772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:47.386816 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:47.387203 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:06:47.387207 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:06:47.387460 (+ 253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:06:47.388974 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:47.388975 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:47.388975 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:47.389132 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:47.389281 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:47.389282 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:47.389282 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:47.389327 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:47.389710 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:06:47.389711 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:06:47.389757 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 00:06:47.389818 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 00:06:47.389938 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":75,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:06:47.415882 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39169
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:06:47.500168 1859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:06:47.500408 1859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:06:47.500434 1859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:06:47.501726 1859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:06:47.501781 1859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:06:47.501803 1859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 00:06:47.501821 1859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:06:47.503437 1859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:06:47.503707 1859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:06:47.503969 1859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:06:47.509402 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:06:47.509531 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:06:47.509394 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:06:47.509649 1859 server_base.cc:1034] running on GCE node
I20260620 00:06:47.509846 1859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:06:47.510155 1859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:06:47.511329 1859 hybrid_clock.cc:648] HybridClock initialized: now 1781914007511312 us; error 28 us; skew 500 ppm
Jun 20 00:06:47 dist-test-slave-grqt krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:06:47.514015 1859 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:06:47.514534 1859 webserver.cc:460] Webserver started at http://127.0.98.3:43997/ using document root <none> and password file <none>
I20260620 00:06:47.514770 1859 fs_manager.cc:359] Metadata directory not provided
I20260620 00:06:47.514842 1859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:06:47.514998 1859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:06:47.516091 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-2/data/instance:
uuid: "4742cdc265e14d38abdc9cd9094182f1"
format_stamp: "Formatted at 2026-06-20 00:06:47 on dist-test-slave-grqt"
server_key: "9dcb71982ca3712ffc5870c3791d8e3b"
server_key_iv: "434d1205e7f0076e1e22f1351ef7b41d"
server_key_version: "encryptionkey@0"
I20260620 00:06:47.516458 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4742cdc265e14d38abdc9cd9094182f1"
format_stamp: "Formatted at 2026-06-20 00:06:47 on dist-test-slave-grqt"
server_key: "9dcb71982ca3712ffc5870c3791d8e3b"
server_key_iv: "434d1205e7f0076e1e22f1351ef7b41d"
server_key_version: "encryptionkey@0"
I20260620 00:06:47.517904 1859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:06:47.518730 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:47.518906 1859 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:06:47.519016 1859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-2/wal
uuid: "4742cdc265e14d38abdc9cd9094182f1"
format_stamp: "Formatted at 2026-06-20 00:06:47 on dist-test-slave-grqt"
server_key: "9dcb71982ca3712ffc5870c3791d8e3b"
server_key_iv: "434d1205e7f0076e1e22f1351ef7b41d"
server_key_version: "encryptionkey@0"
I20260620 00:06:47.519122 1859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:06:47.539207 1859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:06:47.541823 1859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:06:47.541996 1859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:06:47.542605 1859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:06:47.542670 1859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:47.542713 1859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:06:47.542740 1859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 00:06:47 dist-test-slave-grqt krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:06:47.550205 1859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42351
I20260620 00:06:47.550390 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42351 every 8 connection(s)
I20260620 00:06:47.550650 1859 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:06:47.550868 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 1859
I20260620 00:06:47.551268 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:06:47.551476 392 external_mini_cluster.cc:1428] Setting key b7e15bb206895b05d6725ae95337a411
I20260620 00:06:47.553704 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:47.543103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46307 (local address 127.0.98.62:39169)
0620 00:06:47.543173 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:47.543175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:47.543450 (+ 275us) server_negotiation.cc:407] Connection header received
0620 00:06:47.544221 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:47.544223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:47.544238 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:47.544307 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:47.545306 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.545950 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.547684 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.547892 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.547952 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:47.550713 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:47.550714 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:47.550715 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:47.550723 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:47.552483 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:47.552868 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:47.552869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:47.552869 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:47.552921 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:47.553107 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:47.553108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:47.553108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:47.553206 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:47.553257 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:47.553570 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 00:06:47.553637 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 00:06:47.553901 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:47.543077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39169 (local address 127.0.98.3:46307)
0620 00:06:47.543269 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:47.543328 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:47.544084 (+ 756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:47.544402 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:47.544407 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:47.544864 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:06:47.545194 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:47.545202 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.546029 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.546031 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:47.547579 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:47.547583 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.547973 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.547974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:47.548147 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:47.548706 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:06:47.548719 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:06:47.550604 (+ 1885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:06:47.552563 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:47.552568 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:47.552570 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:47.552799 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:47.552963 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:47.552964 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:47.552964 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:47.553058 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:47.553265 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:06:47.553269 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:06:47.553446 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 00:06:47.553554 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 00:06:47.553686 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":102,"threads_started":1}
I20260620 00:06:47.554368 1990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39169
I20260620 00:06:47.554454 1990 heartbeater.cc:464] Registering TS with master...
I20260620 00:06:47.554643 1990 heartbeater.cc:511] Master 127.0.98.62:39169 requested a full tablet report, sending...
I20260620 00:06:47.555030 1506 ts_manager.cc:194] Registered new tserver with Master: 4742cdc265e14d38abdc9cd9094182f1 (127.0.98.3:42351)
I20260620 00:06:47.555527 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46307
I20260620 00:06:47.560998 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:47.555448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39465 (local address 127.0.98.62:39169)
0620 00:06:47.555489 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:47.555491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:47.555564 (+ 73us) server_negotiation.cc:407] Connection header received
0620 00:06:47.555641 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:47.555642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:47.555651 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:47.555716 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:47.556217 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.556700 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.557447 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.557675 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.557743 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:47.558643 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:47.558645 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:47.558645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:47.558651 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:47.560102 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:47.560418 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:47.560419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:47.560419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:47.560474 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:47.560627 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:47.560628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:47.560628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:47.560719 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:47.560764 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:47.560867 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 00:06:47.560924 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 00:06:47.561018 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:47.555383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39169 (local address 127.0.98.3:39465)
0620 00:06:47.555502 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:47.555512 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:47.555575 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:47.555860 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:47.555861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:47.556074 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:06:47.556136 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:47.556138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.556788 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.556788 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:47.557314 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:47.557316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.557750 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.557751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:47.557805 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:47.558258 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:06:47.558262 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:06:47.558574 (+ 312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:06:47.560180 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:47.560181 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:47.560181 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:47.560352 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:47.560529 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:47.560530 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:47.560530 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:47.560587 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:47.560768 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:06:47.560769 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:06:47.560804 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 00:06:47.560853 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 00:06:47.560935 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":70,"threads_started":1}
I20260620 00:06:47.563256 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 00:06:47 dist-test-slave-grqt krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914007, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 00:06:47 dist-test-slave-grqt krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914007, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:06:47.597774 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:47.588911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48776 (local address 127.0.98.62:39169)
0620 00:06:47.588972 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:47.588974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:47.589002 (+ 28us) server_negotiation.cc:407] Connection header received
0620 00:06:47.589041 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:47.589042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:47.589055 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:47.589163 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:47.590062 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.590839 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.592196 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.592404 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.592472 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:47.595097 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:47.595098 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:47.595099 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:47.595106 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:47.596578 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:47.596935 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:47.596936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:47.596936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:47.597000 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:47.597175 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:47.597176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:47.597176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:47.597281 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:47.597356 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:47.597624 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 00:06:47.597703 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 00:06:47.599499 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:06:47.601060 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:06:47.608207 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:47.603648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51372 (local address 127.0.98.2:43591)
0620 00:06:47.603829 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:47.603835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:47.604340 (+ 505us) server_negotiation.cc:407] Connection header received
0620 00:06:47.604462 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:47.604465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:47.604479 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:47.604533 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:06:47.604748 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.605525 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.607104 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.607797 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.607859 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:47.607928 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:47.608066 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 00:06:47.608106 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":117,"threads_started":1}
I20260620 00:06:47.608197 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:47.603479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35335 (local address 127.0.0.1:52774)
0620 00:06:47.603801 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:47.603846 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:47.603915 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:47.604317 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:47.604317 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:47.604319 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:06:47.604366 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:47.604367 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.605682 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.605683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:47.606858 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:47.606861 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.607821 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.607822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:47.607934 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:47.607950 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 00:06:47.608034 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 00:06:47.608108 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":32,"thread_start_us":62,"threads_started":1}
I20260620 00:06:47.608480 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:47.603580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52774 (local address 127.0.98.1:35335)
0620 00:06:47.604017 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:47.604021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:47.604031 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:06:47.604071 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:47.604073 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:47.604092 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:47.604189 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:06:47.604452 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.605589 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.606953 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.607728 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.607815 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:47.607902 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:47.608340 (+ 438us) server_negotiation.cc:299] Negotiation successful
0620 00:06:47.608383 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":108,"threads_started":1}
I20260620 00:06:47.608197 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:47.603480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43591 (local address 127.0.0.1:51372)
0620 00:06:47.604273 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:47.604283 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:47.604369 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:47.604626 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:47.604627 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:47.604628 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:06:47.604677 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:47.604678 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.605608 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.605609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:47.607011 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:47.607015 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.607868 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.607869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:47.607941 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:47.607951 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 00:06:47.608034 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 00:06:47.608092 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"mutex_wait_us":421,"thread_start_us":83,"threads_started":1}
I20260620 00:06:47.609061 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:47.603480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42351 (local address 127.0.0.1:45176)
0620 00:06:47.603945 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:47.603954 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:47.604006 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:47.604215 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:47.604218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:47.604221 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:06:47.604296 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:47.604298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.605651 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.605652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:47.607015 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:47.607016 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.608795 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.608796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:47.608875 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:47.608879 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:06:47.608939 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 00:06:47.608987 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":93,"thread_start_us":74,"threads_started":1}
I20260620 00:06:47.609078 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:47.603737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45176 (local address 127.0.98.3:42351)
0620 00:06:47.603914 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:47.603917 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:47.604017 (+ 100us) server_negotiation.cc:407] Connection header received
0620 00:06:47.604096 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:47.604098 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:47.604112 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:47.604189 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:06:47.604386 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.605581 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.607947 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.608717 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.608785 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:47.608855 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:47.608951 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 00:06:47.608985 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":77,"threads_started":1}
I20260620 00:06:47.609172 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 857ddacce5f3433d8b5953409a8b4081 (DEFAULT_TABLE table=test-table [id=8a19dec80e904ba6863215effedc8eb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:06:47.609580 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 857ddacce5f3433d8b5953409a8b4081. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:06:47.609597 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 857ddacce5f3433d8b5953409a8b4081 (DEFAULT_TABLE table=test-table [id=8a19dec80e904ba6863215effedc8eb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:06:47.609673 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 857ddacce5f3433d8b5953409a8b4081 (DEFAULT_TABLE table=test-table [id=8a19dec80e904ba6863215effedc8eb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:06:47.609925 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 857ddacce5f3433d8b5953409a8b4081. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:06:47.609933 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 857ddacce5f3433d8b5953409a8b4081. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:06:47.612787 2013 tablet_bootstrap.cc:492] T 857ddacce5f3433d8b5953409a8b4081 P 6692ef81cd7a458aaa76cf9c8d53258d: Bootstrap starting.
I20260620 00:06:47.613448 2013 tablet_bootstrap.cc:654] T 857ddacce5f3433d8b5953409a8b4081 P 6692ef81cd7a458aaa76cf9c8d53258d: Neither blocks nor log segments found. Creating new log.
I20260620 00:06:47.613812 2013 log.cc:824] T 857ddacce5f3433d8b5953409a8b4081 P 6692ef81cd7a458aaa76cf9c8d53258d: Log is configured to *not* fsync() on all Append() calls
I20260620 00:06:47.614118 2015 tablet_bootstrap.cc:492] T 857ddacce5f3433d8b5953409a8b4081 P ce6fa384eddd46928bccb5c5d7b5650d: Bootstrap starting.
I20260620 00:06:47.614338 2014 tablet_bootstrap.cc:492] T 857ddacce5f3433d8b5953409a8b4081 P 4742cdc265e14d38abdc9cd9094182f1: Bootstrap starting.
I20260620 00:06:47.614615 2013 tablet_bootstrap.cc:492] T 857ddacce5f3433d8b5953409a8b4081 P 6692ef81cd7a458aaa76cf9c8d53258d: No bootstrap required, opened a new log
I20260620 00:06:47.614701 2013 ts_tablet_manager.cc:1397] T 857ddacce5f3433d8b5953409a8b4081 P 6692ef81cd7a458aaa76cf9c8d53258d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:06:47.614753 2015 tablet_bootstrap.cc:654] T 857ddacce5f3433d8b5953409a8b4081 P ce6fa384eddd46928bccb5c5d7b5650d: Neither blocks nor log segments found. Creating new log.
I20260620 00:06:47.614962 2014 tablet_bootstrap.cc:654] T 857ddacce5f3433d8b5953409a8b4081 P 4742cdc265e14d38abdc9cd9094182f1: Neither blocks nor log segments found. Creating new log.
I20260620 00:06:47.615260 2015 log.cc:824] T 857ddacce5f3433d8b5953409a8b4081 P ce6fa384eddd46928bccb5c5d7b5650d: Log is configured to *not* fsync() on all Append() calls
I20260620 00:06:47.615294 2014 log.cc:824] T 857ddacce5f3433d8b5953409a8b4081 P 4742cdc265e14d38abdc9cd9094182f1: Log is configured to *not* fsync() on all Append() calls
I20260620 00:06:47.616103 2015 tablet_bootstrap.cc:492] T 857ddacce5f3433d8b5953409a8b4081 P ce6fa384eddd46928bccb5c5d7b5650d: No bootstrap required, opened a new log
I20260620 00:06:47.616181 2015 ts_tablet_manager.cc:1397] T 857ddacce5f3433d8b5953409a8b4081 P ce6fa384eddd46928bccb5c5d7b5650d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:06:47.616294 2014 tablet_bootstrap.cc:492] T 857ddacce5f3433d8b5953409a8b4081 P 4742cdc265e14d38abdc9cd9094182f1: No bootstrap required, opened a new log
I20260620 00:06:47.616356 2014 ts_tablet_manager.cc:1397] T 857ddacce5f3433d8b5953409a8b4081 P 4742cdc265e14d38abdc9cd9094182f1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:06:47.616822 2013 raft_consensus.cc:348] T 857ddacce5f3433d8b5953409a8b4081 P 6692ef81cd7a458aaa76cf9c8d53258d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 } } peers { permanent_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35335 } } peers { permanent_uuid: "4742cdc265e14d38abdc9cd9094182f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } }
I20260620 00:06:47.616988 2013 raft_consensus.cc:374] T 857ddacce5f3433d8b5953409a8b4081 P 6692ef81cd7a458aaa76cf9c8d53258d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:06:47.617040 2013 raft_consensus.cc:729] T 857ddacce5f3433d8b5953409a8b4081 P 6692ef81cd7a458aaa76cf9c8d53258d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6692ef81cd7a458aaa76cf9c8d53258d, State: Initialized, Role: FOLLOWER
I20260620 00:06:47.617163 2013 consensus_queue.cc:260] T 857ddacce5f3433d8b5953409a8b4081 P 6692ef81cd7a458aaa76cf9c8d53258d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 } } peers { permanent_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35335 } } peers { permanent_uuid: "4742cdc265e14d38abdc9cd9094182f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } }
I20260620 00:06:47.617488 2013 ts_tablet_manager.cc:1428] T 857ddacce5f3433d8b5953409a8b4081 P 6692ef81cd7a458aaa76cf9c8d53258d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:06:47.617511 1710 heartbeater.cc:503] Master 127.0.98.62:39169 was elected leader, sending a full tablet report...
I20260620 00:06:47.618497 2014 raft_consensus.cc:348] T 857ddacce5f3433d8b5953409a8b4081 P 4742cdc265e14d38abdc9cd9094182f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 } } peers { permanent_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35335 } } peers { permanent_uuid: "4742cdc265e14d38abdc9cd9094182f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } }
I20260620 00:06:47.618680 2014 raft_consensus.cc:374] T 857ddacce5f3433d8b5953409a8b4081 P 4742cdc265e14d38abdc9cd9094182f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:06:47.618729 2014 raft_consensus.cc:729] T 857ddacce5f3433d8b5953409a8b4081 P 4742cdc265e14d38abdc9cd9094182f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4742cdc265e14d38abdc9cd9094182f1, State: Initialized, Role: FOLLOWER
I20260620 00:06:47.618647 2015 raft_consensus.cc:348] T 857ddacce5f3433d8b5953409a8b4081 P ce6fa384eddd46928bccb5c5d7b5650d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 } } peers { permanent_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35335 } } peers { permanent_uuid: "4742cdc265e14d38abdc9cd9094182f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } }
I20260620 00:06:47.618783 2015 raft_consensus.cc:374] T 857ddacce5f3433d8b5953409a8b4081 P ce6fa384eddd46928bccb5c5d7b5650d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:06:47.618824 2015 raft_consensus.cc:729] T 857ddacce5f3433d8b5953409a8b4081 P ce6fa384eddd46928bccb5c5d7b5650d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce6fa384eddd46928bccb5c5d7b5650d, State: Initialized, Role: FOLLOWER
I20260620 00:06:47.618873 2014 consensus_queue.cc:260] T 857ddacce5f3433d8b5953409a8b4081 P 4742cdc265e14d38abdc9cd9094182f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 } } peers { permanent_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35335 } } peers { permanent_uuid: "4742cdc265e14d38abdc9cd9094182f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } }
I20260620 00:06:47.618927 2015 consensus_queue.cc:260] T 857ddacce5f3433d8b5953409a8b4081 P ce6fa384eddd46928bccb5c5d7b5650d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 } } peers { permanent_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35335 } } peers { permanent_uuid: "4742cdc265e14d38abdc9cd9094182f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } }
I20260620 00:06:47.619155 2014 ts_tablet_manager.cc:1428] T 857ddacce5f3433d8b5953409a8b4081 P 4742cdc265e14d38abdc9cd9094182f1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:06:47.619155 2015 ts_tablet_manager.cc:1428] T 857ddacce5f3433d8b5953409a8b4081 P ce6fa384eddd46928bccb5c5d7b5650d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:06:47.619231 1990 heartbeater.cc:503] Master 127.0.98.62:39169 was elected leader, sending a full tablet report...
I20260620 00:06:47.619268 1850 heartbeater.cc:503] Master 127.0.98.62:39169 was elected leader, sending a full tablet report...
W20260620 00:06:47.629927 1851 tablet.cc:2369] T 857ddacce5f3433d8b5953409a8b4081 P ce6fa384eddd46928bccb5c5d7b5650d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 00:06:47.710553 1711 tablet.cc:2369] T 857ddacce5f3433d8b5953409a8b4081 P 6692ef81cd7a458aaa76cf9c8d53258d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 00:06:47.801528 1991 tablet.cc:2369] T 857ddacce5f3433d8b5953409a8b4081 P 4742cdc265e14d38abdc9cd9094182f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:06:47.872629 2019 raft_consensus.cc:482] T 857ddacce5f3433d8b5953409a8b4081 P 6692ef81cd7a458aaa76cf9c8d53258d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:06:47.872777 2019 raft_consensus.cc:504] T 857ddacce5f3433d8b5953409a8b4081 P 6692ef81cd7a458aaa76cf9c8d53258d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 } } peers { permanent_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35335 } } peers { permanent_uuid: "4742cdc265e14d38abdc9cd9094182f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } }
I20260620 00:06:47.873234 2019 leader_election.cc:288] T 857ddacce5f3433d8b5953409a8b4081 P 6692ef81cd7a458aaa76cf9c8d53258d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce6fa384eddd46928bccb5c5d7b5650d (127.0.98.2:43591), 4742cdc265e14d38abdc9cd9094182f1 (127.0.98.3:42351)
I20260620 00:06:47.877444 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:47.873585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44695 (local address 127.0.98.3:42351)
0620 00:06:47.873625 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:47.873627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:47.873679 (+ 52us) server_negotiation.cc:407] Connection header received
0620 00:06:47.873764 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:47.873765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:47.873776 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:47.873839 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:06:47.874054 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.874773 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.876459 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.877025 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.877112 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:47.877124 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:47.877305 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 00:06:47.877346 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 00:06:47.877444 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:47.873578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42031 (local address 127.0.98.2:43591)
0620 00:06:47.873624 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:47.873627 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:47.873679 (+ 52us) server_negotiation.cc:407] Connection header received
0620 00:06:47.873771 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:47.873772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:47.873784 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:47.873839 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:06:47.874021 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.874783 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.876459 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.877052 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.877116 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:47.877124 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:47.877305 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 00:06:47.877346 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 00:06:47.877499 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:47.873495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43591 (local address 127.0.98.1:42031)
0620 00:06:47.873609 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:47.873621 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:47.873692 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:47.873864 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:47.873865 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:47.873867 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:06:47.873936 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:47.873939 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.874875 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.874876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:47.876328 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:47.876331 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.877135 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.877136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:47.877211 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:47.877218 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 00:06:47.877279 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 00:06:47.877339 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":105,"thread_start_us":55,"threads_started":1}
I20260620 00:06:47.877506 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:47.873480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42351 (local address 127.0.98.1:44695)
0620 00:06:47.873609 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:47.873621 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:47.873692 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:47.873864 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:47.873865 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:47.873869 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:06:47.873936 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:47.873939 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.874875 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.874876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:47.876328 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:47.876331 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.877134 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.877135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:47.877213 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:47.877217 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:06:47.877279 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 00:06:47.877339 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":98,"thread_start_us":60,"threads_started":1}
I20260620 00:06:47.877875 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857ddacce5f3433d8b5953409a8b4081" candidate_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4742cdc265e14d38abdc9cd9094182f1" is_pre_election: true
I20260620 00:06:47.877880 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857ddacce5f3433d8b5953409a8b4081" candidate_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" is_pre_election: true
I20260620 00:06:47.878046 1804 raft_consensus.cc:2454] T 857ddacce5f3433d8b5953409a8b4081 P ce6fa384eddd46928bccb5c5d7b5650d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6692ef81cd7a458aaa76cf9c8d53258d in term 0.
I20260620 00:06:47.878082 1944 raft_consensus.cc:2454] T 857ddacce5f3433d8b5953409a8b4081 P 4742cdc265e14d38abdc9cd9094182f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6692ef81cd7a458aaa76cf9c8d53258d in term 0.
I20260620 00:06:47.878327 1596 leader_election.cc:302] T 857ddacce5f3433d8b5953409a8b4081 P 6692ef81cd7a458aaa76cf9c8d53258d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6692ef81cd7a458aaa76cf9c8d53258d, ce6fa384eddd46928bccb5c5d7b5650d; no voters:
I20260620 00:06:47.878453 2019 raft_consensus.cc:2781] T 857ddacce5f3433d8b5953409a8b4081 P 6692ef81cd7a458aaa76cf9c8d53258d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:06:47.878551 2019 raft_consensus.cc:482] T 857ddacce5f3433d8b5953409a8b4081 P 6692ef81cd7a458aaa76cf9c8d53258d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:06:47.878584 2019 raft_consensus.cc:3037] T 857ddacce5f3433d8b5953409a8b4081 P 6692ef81cd7a458aaa76cf9c8d53258d [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:06:47.879326 2019 raft_consensus.cc:504] T 857ddacce5f3433d8b5953409a8b4081 P 6692ef81cd7a458aaa76cf9c8d53258d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 } } peers { permanent_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35335 } } peers { permanent_uuid: "4742cdc265e14d38abdc9cd9094182f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } }
I20260620 00:06:47.879487 2019 leader_election.cc:288] T 857ddacce5f3433d8b5953409a8b4081 P 6692ef81cd7a458aaa76cf9c8d53258d [CANDIDATE]: Term 1 election: Requested vote from peers ce6fa384eddd46928bccb5c5d7b5650d (127.0.98.2:43591), 4742cdc265e14d38abdc9cd9094182f1 (127.0.98.3:42351)
I20260620 00:06:47.879648 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857ddacce5f3433d8b5953409a8b4081" candidate_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4742cdc265e14d38abdc9cd9094182f1"
I20260620 00:06:47.879671 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857ddacce5f3433d8b5953409a8b4081" candidate_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce6fa384eddd46928bccb5c5d7b5650d"
I20260620 00:06:47.879748 1804 raft_consensus.cc:3037] T 857ddacce5f3433d8b5953409a8b4081 P ce6fa384eddd46928bccb5c5d7b5650d [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:06:47.879745 1944 raft_consensus.cc:3037] T 857ddacce5f3433d8b5953409a8b4081 P 4742cdc265e14d38abdc9cd9094182f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:06:47.880473 1944 raft_consensus.cc:2454] T 857ddacce5f3433d8b5953409a8b4081 P 4742cdc265e14d38abdc9cd9094182f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6692ef81cd7a458aaa76cf9c8d53258d in term 1.
I20260620 00:06:47.880615 1804 raft_consensus.cc:2454] T 857ddacce5f3433d8b5953409a8b4081 P ce6fa384eddd46928bccb5c5d7b5650d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6692ef81cd7a458aaa76cf9c8d53258d in term 1.
I20260620 00:06:47.880654 1598 leader_election.cc:302] T 857ddacce5f3433d8b5953409a8b4081 P 6692ef81cd7a458aaa76cf9c8d53258d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4742cdc265e14d38abdc9cd9094182f1, 6692ef81cd7a458aaa76cf9c8d53258d; no voters:
I20260620 00:06:47.880731 2019 raft_consensus.cc:2781] T 857ddacce5f3433d8b5953409a8b4081 P 6692ef81cd7a458aaa76cf9c8d53258d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:06:47.880861 2019 raft_consensus.cc:686] T 857ddacce5f3433d8b5953409a8b4081 P 6692ef81cd7a458aaa76cf9c8d53258d [term 1 LEADER]: Becoming Leader. State: Replica: 6692ef81cd7a458aaa76cf9c8d53258d, State: Running, Role: LEADER
I20260620 00:06:47.880966 2019 consensus_queue.cc:237] T 857ddacce5f3433d8b5953409a8b4081 P 6692ef81cd7a458aaa76cf9c8d53258d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 } } peers { permanent_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35335 } } peers { permanent_uuid: "4742cdc265e14d38abdc9cd9094182f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } }
I20260620 00:06:47.881814 1507 catalog_manager.cc:5495] T 857ddacce5f3433d8b5953409a8b4081 P 6692ef81cd7a458aaa76cf9c8d53258d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6692ef81cd7a458aaa76cf9c8d53258d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4742cdc265e14d38abdc9cd9094182f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } health_report { overall_health: UNKNOWN } } }
I20260620 00:06:47.891623 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:47.889185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52776 (local address 127.0.98.1:35335)
0620 00:06:47.889226 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:47.889228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:47.889245 (+ 17us) server_negotiation.cc:407] Connection header received
0620 00:06:47.889388 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:47.889389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:47.889410 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:47.889473 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:06:47.889630 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.890104 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.890938 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.891119 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.891174 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:47.891345 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:06:47.891437 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:06:47.891494 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:47.891550 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 00:06:47.891567 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 00:06:47.893734 1804 raft_consensus.cc:1261] T 857ddacce5f3433d8b5953409a8b4081 P ce6fa384eddd46928bccb5c5d7b5650d [term 1 FOLLOWER]: Refusing update from remote peer 6692ef81cd7a458aaa76cf9c8d53258d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:06:47.893736 1944 raft_consensus.cc:1261] T 857ddacce5f3433d8b5953409a8b4081 P 4742cdc265e14d38abdc9cd9094182f1 [term 1 FOLLOWER]: Refusing update from remote peer 6692ef81cd7a458aaa76cf9c8d53258d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:06:47.894083 2019 consensus_queue.cc:1035] T 857ddacce5f3433d8b5953409a8b4081 P 6692ef81cd7a458aaa76cf9c8d53258d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4742cdc265e14d38abdc9cd9094182f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:06:47.894241 2024 consensus_queue.cc:1035] T 857ddacce5f3433d8b5953409a8b4081 P 6692ef81cd7a458aaa76cf9c8d53258d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:06:47.896674 2028 mvcc.cc:205] Tried to move back new op lower bound from 7298719776329211904 to 7298719776281153536. Current Snapshot: MvccSnapshot[applied={T|T < 7298719776329211904}]
I20260620 00:06:47.896766 2029 mvcc.cc:205] Tried to move back new op lower bound from 7298719776329211904 to 7298719776281153536. Current Snapshot: MvccSnapshot[applied={T|T < 7298719776329211904}]
I20260620 00:06:47.900951 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:06:47.901072 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:06:47.902006 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3fed10f0e8e94861bc50af06634490e1: Sending DeleteTablet for 3 replicas of tablet 857ddacce5f3433d8b5953409a8b4081
I20260620 00:06:47.902421 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 857ddacce5f3433d8b5953409a8b4081 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:06:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51372
I20260620 00:06:47.902421 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 857ddacce5f3433d8b5953409a8b4081 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:06:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52774
I20260620 00:06:47.902585 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 857ddacce5f3433d8b5953409a8b4081 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:06:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45176
I20260620 00:06:47.902637 2037 tablet_replica.cc:330] T 857ddacce5f3433d8b5953409a8b4081 P ce6fa384eddd46928bccb5c5d7b5650d: stopping tablet replica
I20260620 00:06:47.902742 2037 raft_consensus.cc:2229] T 857ddacce5f3433d8b5953409a8b4081 P ce6fa384eddd46928bccb5c5d7b5650d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:06:47.902854 2038 tablet_replica.cc:330] T 857ddacce5f3433d8b5953409a8b4081 P 6692ef81cd7a458aaa76cf9c8d53258d: stopping tablet replica
I20260620 00:06:47.902873 2037 raft_consensus.cc:2258] T 857ddacce5f3433d8b5953409a8b4081 P ce6fa384eddd46928bccb5c5d7b5650d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:06:47.903019 2038 raft_consensus.cc:2229] T 857ddacce5f3433d8b5953409a8b4081 P 6692ef81cd7a458aaa76cf9c8d53258d [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:06:47.903137 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:06:47.903306 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:06:47.903396 2039 tablet_replica.cc:330] T 857ddacce5f3433d8b5953409a8b4081 P 4742cdc265e14d38abdc9cd9094182f1: stopping tablet replica
I20260620 00:06:47.903515 2039 raft_consensus.cc:2229] T 857ddacce5f3433d8b5953409a8b4081 P 4742cdc265e14d38abdc9cd9094182f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:06:47.903535 2038 raft_consensus.cc:2258] T 857ddacce5f3433d8b5953409a8b4081 P 6692ef81cd7a458aaa76cf9c8d53258d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:06:47.903652 2039 raft_consensus.cc:2258] T 857ddacce5f3433d8b5953409a8b4081 P 4742cdc265e14d38abdc9cd9094182f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:06:47.904156 2038 ts_tablet_manager.cc:1905] T 857ddacce5f3433d8b5953409a8b4081 P 6692ef81cd7a458aaa76cf9c8d53258d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:06:47.904624 2039 ts_tablet_manager.cc:1905] T 857ddacce5f3433d8b5953409a8b4081 P 4742cdc265e14d38abdc9cd9094182f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:06:47.904812 1644 tablet_service.cc:1467] Processing CreateTablet for tablet c3c09941945e4c0ba70c8ef051266f81 (DEFAULT_TABLE table=test-table [id=5ba00415dbad48caa46f27f913babcf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:06:47.904933 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3c09941945e4c0ba70c8ef051266f81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:06:47.906088 2013 tablet_bootstrap.cc:492] T c3c09941945e4c0ba70c8ef051266f81 P 6692ef81cd7a458aaa76cf9c8d53258d: Bootstrap starting.
I20260620 00:06:47.906168 1784 tablet_service.cc:1467] Processing CreateTablet for tablet c3c09941945e4c0ba70c8ef051266f81 (DEFAULT_TABLE table=test-table [id=5ba00415dbad48caa46f27f913babcf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:06:47.906297 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3c09941945e4c0ba70c8ef051266f81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:06:47.906371 2039 ts_tablet_manager.cc:1918] T 857ddacce5f3433d8b5953409a8b4081 P 4742cdc265e14d38abdc9cd9094182f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:06:47.906440 2039 log.cc:1196] T 857ddacce5f3433d8b5953409a8b4081 P 4742cdc265e14d38abdc9cd9094182f1: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-2/wal/wals/857ddacce5f3433d8b5953409a8b4081
I20260620 00:06:47.906517 2013 tablet_bootstrap.cc:654] T c3c09941945e4c0ba70c8ef051266f81 P 6692ef81cd7a458aaa76cf9c8d53258d: Neither blocks nor log segments found. Creating new log.
I20260620 00:06:47.906710 2039 ts_tablet_manager.cc:1939] T 857ddacce5f3433d8b5953409a8b4081 P 4742cdc265e14d38abdc9cd9094182f1: Deleting consensus metadata
I20260620 00:06:47.907073 2038 ts_tablet_manager.cc:1918] T 857ddacce5f3433d8b5953409a8b4081 P 6692ef81cd7a458aaa76cf9c8d53258d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:06:47.907182 2038 log.cc:1196] T 857ddacce5f3433d8b5953409a8b4081 P 6692ef81cd7a458aaa76cf9c8d53258d: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-0/wal/wals/857ddacce5f3433d8b5953409a8b4081
I20260620 00:06:47.907447 2038 ts_tablet_manager.cc:1939] T 857ddacce5f3433d8b5953409a8b4081 P 6692ef81cd7a458aaa76cf9c8d53258d: Deleting consensus metadata
I20260620 00:06:47.907856 2015 tablet_bootstrap.cc:492] T c3c09941945e4c0ba70c8ef051266f81 P ce6fa384eddd46928bccb5c5d7b5650d: Bootstrap starting.
I20260620 00:06:47.908084 2013 tablet_bootstrap.cc:492] T c3c09941945e4c0ba70c8ef051266f81 P 6692ef81cd7a458aaa76cf9c8d53258d: No bootstrap required, opened a new log
I20260620 00:06:47.908082 1491 catalog_manager.cc:4841] TS 6692ef81cd7a458aaa76cf9c8d53258d (127.0.98.1:35335): tablet 857ddacce5f3433d8b5953409a8b4081 (table test-table [id=8a19dec80e904ba6863215effedc8eb6]) successfully deleted
I20260620 00:06:47.908128 2013 ts_tablet_manager.cc:1397] T c3c09941945e4c0ba70c8ef051266f81 P 6692ef81cd7a458aaa76cf9c8d53258d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:06:47.908224 1494 catalog_manager.cc:4841] TS 4742cdc265e14d38abdc9cd9094182f1 (127.0.98.3:42351): tablet 857ddacce5f3433d8b5953409a8b4081 (table test-table [id=8a19dec80e904ba6863215effedc8eb6]) successfully deleted
I20260620 00:06:47.908269 2015 tablet_bootstrap.cc:654] T c3c09941945e4c0ba70c8ef051266f81 P ce6fa384eddd46928bccb5c5d7b5650d: Neither blocks nor log segments found. Creating new log.
I20260620 00:06:47.908272 2013 raft_consensus.cc:348] T c3c09941945e4c0ba70c8ef051266f81 P 6692ef81cd7a458aaa76cf9c8d53258d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35335 } } peers { permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 } } peers { permanent_uuid: "4742cdc265e14d38abdc9cd9094182f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } }
I20260620 00:06:47.908351 2013 raft_consensus.cc:374] T c3c09941945e4c0ba70c8ef051266f81 P 6692ef81cd7a458aaa76cf9c8d53258d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:06:47.908375 2013 raft_consensus.cc:729] T c3c09941945e4c0ba70c8ef051266f81 P 6692ef81cd7a458aaa76cf9c8d53258d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6692ef81cd7a458aaa76cf9c8d53258d, State: Initialized, Role: FOLLOWER
I20260620 00:06:47.908417 2013 consensus_queue.cc:260] T c3c09941945e4c0ba70c8ef051266f81 P 6692ef81cd7a458aaa76cf9c8d53258d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35335 } } peers { permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 } } peers { permanent_uuid: "4742cdc265e14d38abdc9cd9094182f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } }
I20260620 00:06:47.908514 2013 ts_tablet_manager.cc:1428] T c3c09941945e4c0ba70c8ef051266f81 P 6692ef81cd7a458aaa76cf9c8d53258d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:47.908735 1924 tablet_service.cc:1467] Processing CreateTablet for tablet c3c09941945e4c0ba70c8ef051266f81 (DEFAULT_TABLE table=test-table [id=5ba00415dbad48caa46f27f913babcf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:06:47.908831 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3c09941945e4c0ba70c8ef051266f81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:06:47.908816 2037 ts_tablet_manager.cc:1905] T 857ddacce5f3433d8b5953409a8b4081 P ce6fa384eddd46928bccb5c5d7b5650d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:06:47.910077 2014 tablet_bootstrap.cc:492] T c3c09941945e4c0ba70c8ef051266f81 P 4742cdc265e14d38abdc9cd9094182f1: Bootstrap starting.
I20260620 00:06:47.910262 2037 ts_tablet_manager.cc:1918] T 857ddacce5f3433d8b5953409a8b4081 P ce6fa384eddd46928bccb5c5d7b5650d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:06:47.910329 2037 log.cc:1196] T 857ddacce5f3433d8b5953409a8b4081 P ce6fa384eddd46928bccb5c5d7b5650d: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-1/wal/wals/857ddacce5f3433d8b5953409a8b4081
I20260620 00:06:47.910483 2014 tablet_bootstrap.cc:654] T c3c09941945e4c0ba70c8ef051266f81 P 4742cdc265e14d38abdc9cd9094182f1: Neither blocks nor log segments found. Creating new log.
I20260620 00:06:47.910611 2037 ts_tablet_manager.cc:1939] T 857ddacce5f3433d8b5953409a8b4081 P ce6fa384eddd46928bccb5c5d7b5650d: Deleting consensus metadata
I20260620 00:06:47.910758 2015 tablet_bootstrap.cc:492] T c3c09941945e4c0ba70c8ef051266f81 P ce6fa384eddd46928bccb5c5d7b5650d: No bootstrap required, opened a new log
I20260620 00:06:47.910808 2015 ts_tablet_manager.cc:1397] T c3c09941945e4c0ba70c8ef051266f81 P ce6fa384eddd46928bccb5c5d7b5650d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 00:06:47.911049 1492 catalog_manager.cc:4841] TS ce6fa384eddd46928bccb5c5d7b5650d (127.0.98.2:43591): tablet 857ddacce5f3433d8b5953409a8b4081 (table test-table [id=8a19dec80e904ba6863215effedc8eb6]) successfully deleted
I20260620 00:06:47.911564 2015 raft_consensus.cc:348] T c3c09941945e4c0ba70c8ef051266f81 P ce6fa384eddd46928bccb5c5d7b5650d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35335 } } peers { permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 } } peers { permanent_uuid: "4742cdc265e14d38abdc9cd9094182f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } }
I20260620 00:06:47.911652 2015 raft_consensus.cc:374] T c3c09941945e4c0ba70c8ef051266f81 P ce6fa384eddd46928bccb5c5d7b5650d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:06:47.911674 2015 raft_consensus.cc:729] T c3c09941945e4c0ba70c8ef051266f81 P ce6fa384eddd46928bccb5c5d7b5650d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce6fa384eddd46928bccb5c5d7b5650d, State: Initialized, Role: FOLLOWER
I20260620 00:06:47.911718 2015 consensus_queue.cc:260] T c3c09941945e4c0ba70c8ef051266f81 P ce6fa384eddd46928bccb5c5d7b5650d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35335 } } peers { permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 } } peers { permanent_uuid: "4742cdc265e14d38abdc9cd9094182f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } }
I20260620 00:06:47.911811 2015 ts_tablet_manager.cc:1428] T c3c09941945e4c0ba70c8ef051266f81 P ce6fa384eddd46928bccb5c5d7b5650d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:06:47.913797 2014 tablet_bootstrap.cc:492] T c3c09941945e4c0ba70c8ef051266f81 P 4742cdc265e14d38abdc9cd9094182f1: No bootstrap required, opened a new log
I20260620 00:06:47.913854 2014 ts_tablet_manager.cc:1397] T c3c09941945e4c0ba70c8ef051266f81 P 4742cdc265e14d38abdc9cd9094182f1: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 00:06:47.914009 2014 raft_consensus.cc:348] T c3c09941945e4c0ba70c8ef051266f81 P 4742cdc265e14d38abdc9cd9094182f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35335 } } peers { permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 } } peers { permanent_uuid: "4742cdc265e14d38abdc9cd9094182f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } }
I20260620 00:06:47.914085 2014 raft_consensus.cc:374] T c3c09941945e4c0ba70c8ef051266f81 P 4742cdc265e14d38abdc9cd9094182f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:06:47.914106 2014 raft_consensus.cc:729] T c3c09941945e4c0ba70c8ef051266f81 P 4742cdc265e14d38abdc9cd9094182f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4742cdc265e14d38abdc9cd9094182f1, State: Initialized, Role: FOLLOWER
I20260620 00:06:47.914147 2014 consensus_queue.cc:260] T c3c09941945e4c0ba70c8ef051266f81 P 4742cdc265e14d38abdc9cd9094182f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35335 } } peers { permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 } } peers { permanent_uuid: "4742cdc265e14d38abdc9cd9094182f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } }
I20260620 00:06:47.914237 2014 ts_tablet_manager.cc:1428] T c3c09941945e4c0ba70c8ef051266f81 P 4742cdc265e14d38abdc9cd9094182f1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 00:06:47.941803 2020 raft_consensus.cc:482] T c3c09941945e4c0ba70c8ef051266f81 P 4742cdc265e14d38abdc9cd9094182f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:06:47.941937 2020 raft_consensus.cc:504] T c3c09941945e4c0ba70c8ef051266f81 P 4742cdc265e14d38abdc9cd9094182f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35335 } } peers { permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 } } peers { permanent_uuid: "4742cdc265e14d38abdc9cd9094182f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } }
I20260620 00:06:47.942314 2020 leader_election.cc:288] T c3c09941945e4c0ba70c8ef051266f81 P 4742cdc265e14d38abdc9cd9094182f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6692ef81cd7a458aaa76cf9c8d53258d (127.0.98.1:35335), ce6fa384eddd46928bccb5c5d7b5650d (127.0.98.2:43591)
I20260620 00:06:47.945876 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:47.942493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43591 (local address 127.0.98.3:44449)
0620 00:06:47.942559 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:47.942571 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:47.942613 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:47.942900 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:47.942901 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:47.942903 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:06:47.942978 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:47.942980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.944112 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.944113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:47.944917 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:47.944920 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.945560 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.945561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:47.945645 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:47.945648 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:06:47.945723 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 00:06:47.945783 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":48}
I20260620 00:06:47.945858 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:47.942682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44449 (local address 127.0.98.2:43591)
0620 00:06:47.942732 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:47.942734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:47.942743 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:06:47.942772 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:47.942773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:47.942783 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:47.942846 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:06:47.943081 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.944022 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.945015 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.945462 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.945558 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:47.945570 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:47.945732 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 00:06:47.945786 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 00:06:47.946172 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3c09941945e4c0ba70c8ef051266f81" candidate_uuid: "4742cdc265e14d38abdc9cd9094182f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" is_pre_election: true
I20260620 00:06:47.946255 1804 raft_consensus.cc:2454] T c3c09941945e4c0ba70c8ef051266f81 P ce6fa384eddd46928bccb5c5d7b5650d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4742cdc265e14d38abdc9cd9094182f1 in term 0.
I20260620 00:06:47.946445 1876 leader_election.cc:302] T c3c09941945e4c0ba70c8ef051266f81 P 4742cdc265e14d38abdc9cd9094182f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4742cdc265e14d38abdc9cd9094182f1, ce6fa384eddd46928bccb5c5d7b5650d; no voters:
I20260620 00:06:47.946581 2020 raft_consensus.cc:2781] T c3c09941945e4c0ba70c8ef051266f81 P 4742cdc265e14d38abdc9cd9094182f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:06:47.946623 2020 raft_consensus.cc:482] T c3c09941945e4c0ba70c8ef051266f81 P 4742cdc265e14d38abdc9cd9094182f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:06:47.946658 2020 raft_consensus.cc:3037] T c3c09941945e4c0ba70c8ef051266f81 P 4742cdc265e14d38abdc9cd9094182f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:06:47.946980 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:47.942622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51437 (local address 127.0.98.1:35335)
0620 00:06:47.942696 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:47.942698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:47.942763 (+ 65us) server_negotiation.cc:407] Connection header received
0620 00:06:47.942874 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:47.942874 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:47.942884 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:47.942914 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:06:47.943177 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.943881 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.945908 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.946536 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.946590 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:47.946599 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:47.946875 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 00:06:47.946912 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 00:06:47.946988 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:47.942517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35335 (local address 127.0.98.3:51437)
0620 00:06:47.942709 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:47.942717 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:47.942785 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:47.942984 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:47.942985 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:47.942987 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:06:47.943092 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:47.943094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.944008 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.944009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:47.945802 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:47.945804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.946718 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.946719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:47.946797 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:47.946801 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:06:47.946857 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 00:06:47.946895 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":26,"thread_start_us":86,"threads_started":1}
I20260620 00:06:47.947467 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3c09941945e4c0ba70c8ef051266f81" candidate_uuid: "4742cdc265e14d38abdc9cd9094182f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" is_pre_election: true
I20260620 00:06:47.947614 1660 raft_consensus.cc:2454] T c3c09941945e4c0ba70c8ef051266f81 P 6692ef81cd7a458aaa76cf9c8d53258d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4742cdc265e14d38abdc9cd9094182f1 in term 0.
I20260620 00:06:47.947712 2020 raft_consensus.cc:504] T c3c09941945e4c0ba70c8ef051266f81 P 4742cdc265e14d38abdc9cd9094182f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35335 } } peers { permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 } } peers { permanent_uuid: "4742cdc265e14d38abdc9cd9094182f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } }
I20260620 00:06:47.947834 2020 leader_election.cc:288] T c3c09941945e4c0ba70c8ef051266f81 P 4742cdc265e14d38abdc9cd9094182f1 [CANDIDATE]: Term 1 election: Requested vote from peers 6692ef81cd7a458aaa76cf9c8d53258d (127.0.98.1:35335), ce6fa384eddd46928bccb5c5d7b5650d (127.0.98.2:43591)
I20260620 00:06:47.948000 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3c09941945e4c0ba70c8ef051266f81" candidate_uuid: "4742cdc265e14d38abdc9cd9094182f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce6fa384eddd46928bccb5c5d7b5650d"
I20260620 00:06:47.948078 1804 raft_consensus.cc:3037] T c3c09941945e4c0ba70c8ef051266f81 P ce6fa384eddd46928bccb5c5d7b5650d [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:06:47.948709 1804 raft_consensus.cc:2454] T c3c09941945e4c0ba70c8ef051266f81 P ce6fa384eddd46928bccb5c5d7b5650d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4742cdc265e14d38abdc9cd9094182f1 in term 1.
I20260620 00:06:47.948835 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3c09941945e4c0ba70c8ef051266f81" candidate_uuid: "4742cdc265e14d38abdc9cd9094182f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6692ef81cd7a458aaa76cf9c8d53258d"
I20260620 00:06:47.948876 1876 leader_election.cc:302] T c3c09941945e4c0ba70c8ef051266f81 P 4742cdc265e14d38abdc9cd9094182f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4742cdc265e14d38abdc9cd9094182f1, ce6fa384eddd46928bccb5c5d7b5650d; no voters:
I20260620 00:06:47.948907 1660 raft_consensus.cc:3037] T c3c09941945e4c0ba70c8ef051266f81 P 6692ef81cd7a458aaa76cf9c8d53258d [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:06:47.948968 2020 raft_consensus.cc:2781] T c3c09941945e4c0ba70c8ef051266f81 P 4742cdc265e14d38abdc9cd9094182f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:06:47.949120 2020 raft_consensus.cc:686] T c3c09941945e4c0ba70c8ef051266f81 P 4742cdc265e14d38abdc9cd9094182f1 [term 1 LEADER]: Becoming Leader. State: Replica: 4742cdc265e14d38abdc9cd9094182f1, State: Running, Role: LEADER
I20260620 00:06:47.949193 2020 consensus_queue.cc:237] T c3c09941945e4c0ba70c8ef051266f81 P 4742cdc265e14d38abdc9cd9094182f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35335 } } peers { permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 } } peers { permanent_uuid: "4742cdc265e14d38abdc9cd9094182f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } }
I20260620 00:06:47.949762 1660 raft_consensus.cc:2454] T c3c09941945e4c0ba70c8ef051266f81 P 6692ef81cd7a458aaa76cf9c8d53258d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4742cdc265e14d38abdc9cd9094182f1 in term 1.
I20260620 00:06:47.949855 1506 catalog_manager.cc:5495] T c3c09941945e4c0ba70c8ef051266f81 P 4742cdc265e14d38abdc9cd9094182f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4742cdc265e14d38abdc9cd9094182f1 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "4742cdc265e14d38abdc9cd9094182f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4742cdc265e14d38abdc9cd9094182f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } health_report { overall_health: HEALTHY } } }
Jun 20 00:06:47 dist-test-slave-grqt krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:06:47.989666 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:47.978343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48790 (local address 127.0.98.62:39169)
0620 00:06:47.978394 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:47.978397 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:47.978585 (+ 188us) server_negotiation.cc:407] Connection header received
0620 00:06:47.978621 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:47.978622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:47.978635 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:47.978706 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:47.979333 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.979809 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.980847 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.981022 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.981080 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:47.986799 (+ 5719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:47.986801 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:47.986801 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:47.986815 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:47.988321 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:47.988775 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:47.988776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:47.988777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:47.988846 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:47.989074 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:47.989075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:47.989075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:47.989200 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:47.989281 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:47.989458 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 00:06:47.989551 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 00:06:47.989737 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:47.978249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39169 (local address 127.0.0.1:48790)
0620 00:06:47.978437 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:47.978450 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:47.978538 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:47.978735 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:47.978736 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:47.979183 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:06:47.979242 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:47.979245 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.979898 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.979899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:47.980764 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:47.980767 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:47.981110 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:47.981111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:47.981175 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:47.984969 (+ 3794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:06:47.984977 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:06:47.986674 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:06:47.988449 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:47.988458 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:47.988461 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:47.988682 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:47.988916 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:47.988916 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:47.988917 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:47.988984 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:47.989270 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:06:47.989278 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:06:47.989329 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 00:06:47.989436 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 00:06:47.989612 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":63,"thread_start_us":98,"threads_started":1}
I20260620 00:06:47.992519 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48790:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 00:06:47.992741 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:06:47.994503 1784 tablet_service.cc:1467] Processing CreateTablet for tablet cc0c5f3f97304e34a2fbead2ad16292a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=52d3204e2d044226af1da7c9f337b9fd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 00:06:47.994549 1644 tablet_service.cc:1467] Processing CreateTablet for tablet cc0c5f3f97304e34a2fbead2ad16292a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=52d3204e2d044226af1da7c9f337b9fd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 00:06:47.994707 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc0c5f3f97304e34a2fbead2ad16292a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:06:47.994735 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc0c5f3f97304e34a2fbead2ad16292a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:06:47.996006 2013 tablet_bootstrap.cc:492] T cc0c5f3f97304e34a2fbead2ad16292a P 6692ef81cd7a458aaa76cf9c8d53258d: Bootstrap starting.
I20260620 00:06:47.996491 2013 tablet_bootstrap.cc:654] T cc0c5f3f97304e34a2fbead2ad16292a P 6692ef81cd7a458aaa76cf9c8d53258d: Neither blocks nor log segments found. Creating new log.
I20260620 00:06:47.996479 1924 tablet_service.cc:1467] Processing CreateTablet for tablet cc0c5f3f97304e34a2fbead2ad16292a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=52d3204e2d044226af1da7c9f337b9fd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 00:06:47.996636 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc0c5f3f97304e34a2fbead2ad16292a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:06:47.997123 2013 tablet_bootstrap.cc:492] T cc0c5f3f97304e34a2fbead2ad16292a P 6692ef81cd7a458aaa76cf9c8d53258d: No bootstrap required, opened a new log
I20260620 00:06:47.997187 2013 ts_tablet_manager.cc:1397] T cc0c5f3f97304e34a2fbead2ad16292a P 6692ef81cd7a458aaa76cf9c8d53258d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:06:47.997345 2013 raft_consensus.cc:348] T cc0c5f3f97304e34a2fbead2ad16292a P 6692ef81cd7a458aaa76cf9c8d53258d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35335 } } peers { permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 } } peers { permanent_uuid: "4742cdc265e14d38abdc9cd9094182f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } }
I20260620 00:06:47.997458 2015 tablet_bootstrap.cc:492] T cc0c5f3f97304e34a2fbead2ad16292a P ce6fa384eddd46928bccb5c5d7b5650d: Bootstrap starting.
I20260620 00:06:47.997486 2013 raft_consensus.cc:374] T cc0c5f3f97304e34a2fbead2ad16292a P 6692ef81cd7a458aaa76cf9c8d53258d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:06:47.997510 2013 raft_consensus.cc:729] T cc0c5f3f97304e34a2fbead2ad16292a P 6692ef81cd7a458aaa76cf9c8d53258d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6692ef81cd7a458aaa76cf9c8d53258d, State: Initialized, Role: FOLLOWER
I20260620 00:06:47.997550 2013 consensus_queue.cc:260] T cc0c5f3f97304e34a2fbead2ad16292a P 6692ef81cd7a458aaa76cf9c8d53258d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35335 } } peers { permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 } } peers { permanent_uuid: "4742cdc265e14d38abdc9cd9094182f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } }
I20260620 00:06:47.998051 2013 ts_tablet_manager.cc:1428] T cc0c5f3f97304e34a2fbead2ad16292a P 6692ef81cd7a458aaa76cf9c8d53258d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:06:47.998174 2032 tablet_replica.cc:439] T cc0c5f3f97304e34a2fbead2ad16292a P 6692ef81cd7a458aaa76cf9c8d53258d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35335 } } peers { permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 } } peers { permanent_uuid: "4742cdc265e14d38abdc9cd9094182f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } } }
I20260620 00:06:47.998242 2032 tablet_replica.cc:442] T cc0c5f3f97304e34a2fbead2ad16292a P 6692ef81cd7a458aaa76cf9c8d53258d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:06:47.998298 2014 tablet_bootstrap.cc:492] T cc0c5f3f97304e34a2fbead2ad16292a P 4742cdc265e14d38abdc9cd9094182f1: Bootstrap starting.
I20260620 00:06:47.998345 2015 tablet_bootstrap.cc:654] T cc0c5f3f97304e34a2fbead2ad16292a P ce6fa384eddd46928bccb5c5d7b5650d: Neither blocks nor log segments found. Creating new log.
I20260620 00:06:47.998737 2014 tablet_bootstrap.cc:654] T cc0c5f3f97304e34a2fbead2ad16292a P 4742cdc265e14d38abdc9cd9094182f1: Neither blocks nor log segments found. Creating new log.
I20260620 00:06:47.998945 2015 tablet_bootstrap.cc:492] T cc0c5f3f97304e34a2fbead2ad16292a P ce6fa384eddd46928bccb5c5d7b5650d: No bootstrap required, opened a new log
I20260620 00:06:47.999027 2015 ts_tablet_manager.cc:1397] T cc0c5f3f97304e34a2fbead2ad16292a P ce6fa384eddd46928bccb5c5d7b5650d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:06:47.999179 2015 raft_consensus.cc:348] T cc0c5f3f97304e34a2fbead2ad16292a P ce6fa384eddd46928bccb5c5d7b5650d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35335 } } peers { permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 } } peers { permanent_uuid: "4742cdc265e14d38abdc9cd9094182f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } }
I20260620 00:06:47.999244 2015 raft_consensus.cc:374] T cc0c5f3f97304e34a2fbead2ad16292a P ce6fa384eddd46928bccb5c5d7b5650d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:06:47.999264 2015 raft_consensus.cc:729] T cc0c5f3f97304e34a2fbead2ad16292a P ce6fa384eddd46928bccb5c5d7b5650d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce6fa384eddd46928bccb5c5d7b5650d, State: Initialized, Role: FOLLOWER
I20260620 00:06:47.999307 2015 consensus_queue.cc:260] T cc0c5f3f97304e34a2fbead2ad16292a P ce6fa384eddd46928bccb5c5d7b5650d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35335 } } peers { permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 } } peers { permanent_uuid: "4742cdc265e14d38abdc9cd9094182f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } }
I20260620 00:06:47.999540 2015 ts_tablet_manager.cc:1428] T cc0c5f3f97304e34a2fbead2ad16292a P ce6fa384eddd46928bccb5c5d7b5650d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:47.999645 2014 tablet_bootstrap.cc:492] T cc0c5f3f97304e34a2fbead2ad16292a P 4742cdc265e14d38abdc9cd9094182f1: No bootstrap required, opened a new log
I20260620 00:06:47.999704 2014 ts_tablet_manager.cc:1397] T cc0c5f3f97304e34a2fbead2ad16292a P 4742cdc265e14d38abdc9cd9094182f1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:06:48.000029 2014 raft_consensus.cc:348] T cc0c5f3f97304e34a2fbead2ad16292a P 4742cdc265e14d38abdc9cd9094182f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35335 } } peers { permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 } } peers { permanent_uuid: "4742cdc265e14d38abdc9cd9094182f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } }
I20260620 00:06:48.000128 2014 raft_consensus.cc:374] T cc0c5f3f97304e34a2fbead2ad16292a P 4742cdc265e14d38abdc9cd9094182f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:06:48.000149 2014 raft_consensus.cc:729] T cc0c5f3f97304e34a2fbead2ad16292a P 4742cdc265e14d38abdc9cd9094182f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4742cdc265e14d38abdc9cd9094182f1, State: Initialized, Role: FOLLOWER
I20260620 00:06:48.000190 2014 consensus_queue.cc:260] T cc0c5f3f97304e34a2fbead2ad16292a P 4742cdc265e14d38abdc9cd9094182f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35335 } } peers { permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 } } peers { permanent_uuid: "4742cdc265e14d38abdc9cd9094182f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } }
I20260620 00:06:48.000276 2014 ts_tablet_manager.cc:1428] T cc0c5f3f97304e34a2fbead2ad16292a P 4742cdc265e14d38abdc9cd9094182f1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:06:48.000443 2021 tablet_replica.cc:439] T cc0c5f3f97304e34a2fbead2ad16292a P ce6fa384eddd46928bccb5c5d7b5650d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35335 } } peers { permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 } } peers { permanent_uuid: "4742cdc265e14d38abdc9cd9094182f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } } }
I20260620 00:06:48.000542 2021 tablet_replica.cc:442] T cc0c5f3f97304e34a2fbead2ad16292a P ce6fa384eddd46928bccb5c5d7b5650d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:06:48.000761 2020 tablet_replica.cc:439] T cc0c5f3f97304e34a2fbead2ad16292a P 4742cdc265e14d38abdc9cd9094182f1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35335 } } peers { permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 } } peers { permanent_uuid: "4742cdc265e14d38abdc9cd9094182f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } } }
I20260620 00:06:48.000855 2020 tablet_replica.cc:442] T cc0c5f3f97304e34a2fbead2ad16292a P 4742cdc265e14d38abdc9cd9094182f1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:06:48.172978 2020 raft_consensus.cc:482] T cc0c5f3f97304e34a2fbead2ad16292a P 4742cdc265e14d38abdc9cd9094182f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:06:48.173105 2020 raft_consensus.cc:504] T cc0c5f3f97304e34a2fbead2ad16292a P 4742cdc265e14d38abdc9cd9094182f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35335 } } peers { permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 } } peers { permanent_uuid: "4742cdc265e14d38abdc9cd9094182f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } }
I20260620 00:06:48.173282 2020 leader_election.cc:288] T cc0c5f3f97304e34a2fbead2ad16292a P 4742cdc265e14d38abdc9cd9094182f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6692ef81cd7a458aaa76cf9c8d53258d (127.0.98.1:35335), ce6fa384eddd46928bccb5c5d7b5650d (127.0.98.2:43591)
I20260620 00:06:48.173508 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc0c5f3f97304e34a2fbead2ad16292a" candidate_uuid: "4742cdc265e14d38abdc9cd9094182f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" is_pre_election: true
I20260620 00:06:48.173558 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc0c5f3f97304e34a2fbead2ad16292a" candidate_uuid: "4742cdc265e14d38abdc9cd9094182f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" is_pre_election: true
I20260620 00:06:48.173626 1660 raft_consensus.cc:2454] T cc0c5f3f97304e34a2fbead2ad16292a P 6692ef81cd7a458aaa76cf9c8d53258d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4742cdc265e14d38abdc9cd9094182f1 in term 0.
I20260620 00:06:48.173666 1804 raft_consensus.cc:2454] T cc0c5f3f97304e34a2fbead2ad16292a P ce6fa384eddd46928bccb5c5d7b5650d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4742cdc265e14d38abdc9cd9094182f1 in term 0.
I20260620 00:06:48.173790 1875 leader_election.cc:302] T cc0c5f3f97304e34a2fbead2ad16292a P 4742cdc265e14d38abdc9cd9094182f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4742cdc265e14d38abdc9cd9094182f1, 6692ef81cd7a458aaa76cf9c8d53258d; no voters:
I20260620 00:06:48.173900 2020 raft_consensus.cc:2781] T cc0c5f3f97304e34a2fbead2ad16292a P 4742cdc265e14d38abdc9cd9094182f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:06:48.173954 2020 raft_consensus.cc:482] T cc0c5f3f97304e34a2fbead2ad16292a P 4742cdc265e14d38abdc9cd9094182f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:06:48.174013 2020 raft_consensus.cc:3037] T cc0c5f3f97304e34a2fbead2ad16292a P 4742cdc265e14d38abdc9cd9094182f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:06:48.174690 2020 raft_consensus.cc:504] T cc0c5f3f97304e34a2fbead2ad16292a P 4742cdc265e14d38abdc9cd9094182f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35335 } } peers { permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 } } peers { permanent_uuid: "4742cdc265e14d38abdc9cd9094182f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } }
I20260620 00:06:48.174835 2020 leader_election.cc:288] T cc0c5f3f97304e34a2fbead2ad16292a P 4742cdc265e14d38abdc9cd9094182f1 [CANDIDATE]: Term 1 election: Requested vote from peers 6692ef81cd7a458aaa76cf9c8d53258d (127.0.98.1:35335), ce6fa384eddd46928bccb5c5d7b5650d (127.0.98.2:43591)
I20260620 00:06:48.174993 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc0c5f3f97304e34a2fbead2ad16292a" candidate_uuid: "4742cdc265e14d38abdc9cd9094182f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6692ef81cd7a458aaa76cf9c8d53258d"
I20260620 00:06:48.175005 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc0c5f3f97304e34a2fbead2ad16292a" candidate_uuid: "4742cdc265e14d38abdc9cd9094182f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce6fa384eddd46928bccb5c5d7b5650d"
I20260620 00:06:48.175079 1660 raft_consensus.cc:3037] T cc0c5f3f97304e34a2fbead2ad16292a P 6692ef81cd7a458aaa76cf9c8d53258d [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:06:48.175076 1804 raft_consensus.cc:3037] T cc0c5f3f97304e34a2fbead2ad16292a P ce6fa384eddd46928bccb5c5d7b5650d [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:06:48.175619 1660 raft_consensus.cc:2454] T cc0c5f3f97304e34a2fbead2ad16292a P 6692ef81cd7a458aaa76cf9c8d53258d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4742cdc265e14d38abdc9cd9094182f1 in term 1.
I20260620 00:06:48.175788 1875 leader_election.cc:302] T cc0c5f3f97304e34a2fbead2ad16292a P 4742cdc265e14d38abdc9cd9094182f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4742cdc265e14d38abdc9cd9094182f1, 6692ef81cd7a458aaa76cf9c8d53258d; no voters:
I20260620 00:06:48.175786 1804 raft_consensus.cc:2454] T cc0c5f3f97304e34a2fbead2ad16292a P ce6fa384eddd46928bccb5c5d7b5650d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4742cdc265e14d38abdc9cd9094182f1 in term 1.
I20260620 00:06:48.175890 2020 raft_consensus.cc:2781] T cc0c5f3f97304e34a2fbead2ad16292a P 4742cdc265e14d38abdc9cd9094182f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:06:48.175951 2020 raft_consensus.cc:686] T cc0c5f3f97304e34a2fbead2ad16292a P 4742cdc265e14d38abdc9cd9094182f1 [term 1 LEADER]: Becoming Leader. State: Replica: 4742cdc265e14d38abdc9cd9094182f1, State: Running, Role: LEADER
I20260620 00:06:48.176000 2020 consensus_queue.cc:237] T cc0c5f3f97304e34a2fbead2ad16292a P 4742cdc265e14d38abdc9cd9094182f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35335 } } peers { permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 } } peers { permanent_uuid: "4742cdc265e14d38abdc9cd9094182f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } }
I20260620 00:06:48.176307 2044 tablet_replica.cc:439] T cc0c5f3f97304e34a2fbead2ad16292a P 4742cdc265e14d38abdc9cd9094182f1: TxnStatusTablet state changed. Reason: New leader 4742cdc265e14d38abdc9cd9094182f1. Latest consensus state: current_term: 1 leader_uuid: "4742cdc265e14d38abdc9cd9094182f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35335 } } peers { permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 } } peers { permanent_uuid: "4742cdc265e14d38abdc9cd9094182f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } } }
I20260620 00:06:48.176400 2044 tablet_replica.cc:442] T cc0c5f3f97304e34a2fbead2ad16292a P 4742cdc265e14d38abdc9cd9094182f1: This TxnStatusTablet replica's current role is: LEADER
I20260620 00:06:48.176554 2056 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 00:06:48.176683 2056 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 00:06:48.176704 1507 catalog_manager.cc:5495] T cc0c5f3f97304e34a2fbead2ad16292a P 4742cdc265e14d38abdc9cd9094182f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4742cdc265e14d38abdc9cd9094182f1 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "4742cdc265e14d38abdc9cd9094182f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4742cdc265e14d38abdc9cd9094182f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } health_report { overall_health: HEALTHY } } }
I20260620 00:06:48.186619 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:48.183281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42351 (local address 127.0.0.1:45190)
0620 00:06:48.183361 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:48.183379 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:48.183431 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:48.183837 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:48.183838 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:48.183840 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 00:06:48.183890 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:48.183892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:48.184850 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:48.184851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:48.185496 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:48.185499 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:48.186108 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:48.186109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:48.186165 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:48.186180 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 00:06:48.186468 (+ 288us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 00:06:48.186475 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 00:06:48.186514 (+ 39us) client_negotiation.cc:240] Negotiation successful
0620 00:06:48.186558 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 00:06:48.186621 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:48.183507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45190 (local address 127.0.98.3:42351)
0620 00:06:48.183572 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:48.183574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:48.183584 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:06:48.183615 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:48.183615 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:48.183634 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:48.183715 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:06:48.183970 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:48.184753 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:48.185570 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:48.185871 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:48.185946 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:48.186280 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:06:48.186398 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:06:48.186475 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:48.186521 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 00:06:48.186542 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 00:06:48.187855 1804 raft_consensus.cc:1261] T cc0c5f3f97304e34a2fbead2ad16292a P ce6fa384eddd46928bccb5c5d7b5650d [term 1 FOLLOWER]: Refusing update from remote peer 4742cdc265e14d38abdc9cd9094182f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:06:48.187875 1660 raft_consensus.cc:1261] T cc0c5f3f97304e34a2fbead2ad16292a P 6692ef81cd7a458aaa76cf9c8d53258d [term 1 FOLLOWER]: Refusing update from remote peer 4742cdc265e14d38abdc9cd9094182f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:06:48.188298 2060 consensus_queue.cc:1035] T cc0c5f3f97304e34a2fbead2ad16292a P 4742cdc265e14d38abdc9cd9094182f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:06:48.188432 2044 consensus_queue.cc:1035] T cc0c5f3f97304e34a2fbead2ad16292a P 4742cdc265e14d38abdc9cd9094182f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:06:48.189042 2032 tablet_replica.cc:439] T cc0c5f3f97304e34a2fbead2ad16292a P 6692ef81cd7a458aaa76cf9c8d53258d: TxnStatusTablet state changed. Reason: New leader 4742cdc265e14d38abdc9cd9094182f1. Latest consensus state: current_term: 1 leader_uuid: "4742cdc265e14d38abdc9cd9094182f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35335 } } peers { permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 } } peers { permanent_uuid: "4742cdc265e14d38abdc9cd9094182f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } } }
I20260620 00:06:48.189136 2032 tablet_replica.cc:442] T cc0c5f3f97304e34a2fbead2ad16292a P 6692ef81cd7a458aaa76cf9c8d53258d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:06:48.189601 2021 tablet_replica.cc:439] T cc0c5f3f97304e34a2fbead2ad16292a P ce6fa384eddd46928bccb5c5d7b5650d: TxnStatusTablet state changed. Reason: New leader 4742cdc265e14d38abdc9cd9094182f1. Latest consensus state: current_term: 1 leader_uuid: "4742cdc265e14d38abdc9cd9094182f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35335 } } peers { permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 } } peers { permanent_uuid: "4742cdc265e14d38abdc9cd9094182f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } } }
I20260620 00:06:48.189693 2021 tablet_replica.cc:442] T cc0c5f3f97304e34a2fbead2ad16292a P ce6fa384eddd46928bccb5c5d7b5650d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:06:48.189618 2044 tablet_replica.cc:439] T cc0c5f3f97304e34a2fbead2ad16292a P 4742cdc265e14d38abdc9cd9094182f1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4742cdc265e14d38abdc9cd9094182f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35335 } } peers { permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 } } peers { permanent_uuid: "4742cdc265e14d38abdc9cd9094182f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } } }
I20260620 00:06:48.189723 2044 tablet_replica.cc:442] T cc0c5f3f97304e34a2fbead2ad16292a P 4742cdc265e14d38abdc9cd9094182f1: This TxnStatusTablet replica's current role is: LEADER
I20260620 00:06:48.190050 2027 mvcc.cc:205] Tried to move back new op lower bound from 7298719777535266816 to 7298719777489375232. Current Snapshot: MvccSnapshot[applied={T|T < 7298719777535266816}]
I20260620 00:06:48.190251 2032 tablet_replica.cc:439] T cc0c5f3f97304e34a2fbead2ad16292a P 6692ef81cd7a458aaa76cf9c8d53258d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4742cdc265e14d38abdc9cd9094182f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35335 } } peers { permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 } } peers { permanent_uuid: "4742cdc265e14d38abdc9cd9094182f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } } }
I20260620 00:06:48.190337 2032 tablet_replica.cc:442] T cc0c5f3f97304e34a2fbead2ad16292a P 6692ef81cd7a458aaa76cf9c8d53258d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:06:48.191375 2044 tablet_replica.cc:439] T cc0c5f3f97304e34a2fbead2ad16292a P 4742cdc265e14d38abdc9cd9094182f1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4742cdc265e14d38abdc9cd9094182f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35335 } } peers { permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 } } peers { permanent_uuid: "4742cdc265e14d38abdc9cd9094182f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } } }
I20260620 00:06:48.191455 2044 tablet_replica.cc:442] T cc0c5f3f97304e34a2fbead2ad16292a P 4742cdc265e14d38abdc9cd9094182f1: This TxnStatusTablet replica's current role is: LEADER
I20260620 00:06:48.192401 2021 tablet_replica.cc:439] T cc0c5f3f97304e34a2fbead2ad16292a P ce6fa384eddd46928bccb5c5d7b5650d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4742cdc265e14d38abdc9cd9094182f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35335 } } peers { permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 } } peers { permanent_uuid: "4742cdc265e14d38abdc9cd9094182f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } } }
I20260620 00:06:48.192494 2021 tablet_replica.cc:442] T cc0c5f3f97304e34a2fbead2ad16292a P ce6fa384eddd46928bccb5c5d7b5650d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:06:48.195174 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:48.191726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45204 (local address 127.0.98.3:42351)
0620 00:06:48.192505 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:48.192507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:48.192513 (+ 6us) server_negotiation.cc:407] Connection header received
0620 00:06:48.192533 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:48.192534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:48.192545 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:48.192599 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:06:48.192746 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:48.193522 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:48.194589 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:48.194771 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:48.194821 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:48.194972 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:06:48.194999 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:06:48.195040 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:48.195108 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 00:06:48.195124 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":754}
I20260620 00:06:48.200348 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:48.197596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56905 (local address 127.0.98.3:42351)
0620 00:06:48.197651 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:48.197653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:48.197789 (+ 136us) server_negotiation.cc:407] Connection header received
0620 00:06:48.197852 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:48.197853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:48.197860 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:48.197899 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:06:48.198023 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:48.198451 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:48.199660 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:48.199850 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:48.199911 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:48.200115 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:06:48.200139 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:06:48.200189 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:48.200255 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 00:06:48.200275 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":34}
I20260620 00:06:48.200448 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:48.197540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42351 (local address 127.0.98.3:56905)
0620 00:06:48.197717 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:48.197728 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:48.197789 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:48.197911 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:48.197911 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:48.197913 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 00:06:48.197969 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:48.197970 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:48.198530 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:48.198531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:48.199560 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:48.199562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:48.199954 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:48.199956 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:48.200020 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:48.200035 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 00:06:48.200198 (+ 163us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 00:06:48.200200 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 00:06:48.200253 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 00:06:48.200302 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":1,"thread_start_us":105,"threads_started":1}
I20260620 00:06:48.202828 1660 raft_consensus.cc:1261] T c3c09941945e4c0ba70c8ef051266f81 P 6692ef81cd7a458aaa76cf9c8d53258d [term 1 FOLLOWER]: Refusing update from remote peer 4742cdc265e14d38abdc9cd9094182f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:06:48.202859 1804 raft_consensus.cc:1261] T c3c09941945e4c0ba70c8ef051266f81 P ce6fa384eddd46928bccb5c5d7b5650d [term 1 FOLLOWER]: Refusing update from remote peer 4742cdc265e14d38abdc9cd9094182f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:06:48.203164 2044 consensus_queue.cc:1035] T c3c09941945e4c0ba70c8ef051266f81 P 4742cdc265e14d38abdc9cd9094182f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6692ef81cd7a458aaa76cf9c8d53258d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:06:48.203270 2059 consensus_queue.cc:1035] T c3c09941945e4c0ba70c8ef051266f81 P 4742cdc265e14d38abdc9cd9094182f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce6fa384eddd46928bccb5c5d7b5650d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:06:48.218032 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:06:48.218156 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:06:48.218766 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3fed10f0e8e94861bc50af06634490e1: Sending DeleteTablet for 3 replicas of tablet c3c09941945e4c0ba70c8ef051266f81
I20260620 00:06:48.219024 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet c3c09941945e4c0ba70c8ef051266f81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:06:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52774
I20260620 00:06:48.219022 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet c3c09941945e4c0ba70c8ef051266f81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:06:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51372
I20260620 00:06:48.219138 2038 tablet_replica.cc:330] T c3c09941945e4c0ba70c8ef051266f81 P 6692ef81cd7a458aaa76cf9c8d53258d: stopping tablet replica
I20260620 00:06:48.219163 2037 tablet_replica.cc:330] T c3c09941945e4c0ba70c8ef051266f81 P ce6fa384eddd46928bccb5c5d7b5650d: stopping tablet replica
I20260620 00:06:48.219218 2038 raft_consensus.cc:2229] T c3c09941945e4c0ba70c8ef051266f81 P 6692ef81cd7a458aaa76cf9c8d53258d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:06:48.219218 2037 raft_consensus.cc:2229] T c3c09941945e4c0ba70c8ef051266f81 P ce6fa384eddd46928bccb5c5d7b5650d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:06:48.219254 2038 raft_consensus.cc:2258] T c3c09941945e4c0ba70c8ef051266f81 P 6692ef81cd7a458aaa76cf9c8d53258d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:06:48.219254 2037 raft_consensus.cc:2258] T c3c09941945e4c0ba70c8ef051266f81 P ce6fa384eddd46928bccb5c5d7b5650d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:06:48.219236 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet c3c09941945e4c0ba70c8ef051266f81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:06:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45176
I20260620 00:06:48.219343 2039 tablet_replica.cc:330] T c3c09941945e4c0ba70c8ef051266f81 P 4742cdc265e14d38abdc9cd9094182f1: stopping tablet replica
I20260620 00:06:48.219398 2039 raft_consensus.cc:2229] T c3c09941945e4c0ba70c8ef051266f81 P 4742cdc265e14d38abdc9cd9094182f1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:06:48.219491 2039 raft_consensus.cc:2258] T c3c09941945e4c0ba70c8ef051266f81 P 4742cdc265e14d38abdc9cd9094182f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:06:48.219544 2037 ts_tablet_manager.cc:1905] T c3c09941945e4c0ba70c8ef051266f81 P ce6fa384eddd46928bccb5c5d7b5650d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:06:48.219761 2039 ts_tablet_manager.cc:1905] T c3c09941945e4c0ba70c8ef051266f81 P 4742cdc265e14d38abdc9cd9094182f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:06:48.219847 2038 ts_tablet_manager.cc:1905] T c3c09941945e4c0ba70c8ef051266f81 P 6692ef81cd7a458aaa76cf9c8d53258d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:06:48.221143 2037 ts_tablet_manager.cc:1918] T c3c09941945e4c0ba70c8ef051266f81 P ce6fa384eddd46928bccb5c5d7b5650d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 00:06:48.221168 2038 ts_tablet_manager.cc:1918] T c3c09941945e4c0ba70c8ef051266f81 P 6692ef81cd7a458aaa76cf9c8d53258d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 00:06:48.221205 2037 log.cc:1196] T c3c09941945e4c0ba70c8ef051266f81 P ce6fa384eddd46928bccb5c5d7b5650d: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-1/wal/wals/c3c09941945e4c0ba70c8ef051266f81
I20260620 00:06:48.221217 2038 log.cc:1196] T c3c09941945e4c0ba70c8ef051266f81 P 6692ef81cd7a458aaa76cf9c8d53258d: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-0/wal/wals/c3c09941945e4c0ba70c8ef051266f81
I20260620 00:06:48.221230 2039 ts_tablet_manager.cc:1918] T c3c09941945e4c0ba70c8ef051266f81 P 4742cdc265e14d38abdc9cd9094182f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 00:06:48.221278 2039 log.cc:1196] T c3c09941945e4c0ba70c8ef051266f81 P 4742cdc265e14d38abdc9cd9094182f1: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914001714546-392-0/minicluster-data/ts-2/wal/wals/c3c09941945e4c0ba70c8ef051266f81
I20260620 00:06:48.221482 2037 ts_tablet_manager.cc:1939] T c3c09941945e4c0ba70c8ef051266f81 P ce6fa384eddd46928bccb5c5d7b5650d: Deleting consensus metadata
I20260620 00:06:48.221524 2039 ts_tablet_manager.cc:1939] T c3c09941945e4c0ba70c8ef051266f81 P 4742cdc265e14d38abdc9cd9094182f1: Deleting consensus metadata
I20260620 00:06:48.221696 2038 ts_tablet_manager.cc:1939] T c3c09941945e4c0ba70c8ef051266f81 P 6692ef81cd7a458aaa76cf9c8d53258d: Deleting consensus metadata
I20260620 00:06:48.221812 1492 catalog_manager.cc:4841] TS ce6fa384eddd46928bccb5c5d7b5650d (127.0.98.2:43591): tablet c3c09941945e4c0ba70c8ef051266f81 (table test-table [id=5ba00415dbad48caa46f27f913babcf2]) successfully deleted
I20260620 00:06:48.221977 1491 catalog_manager.cc:4841] TS 6692ef81cd7a458aaa76cf9c8d53258d (127.0.98.1:35335): tablet c3c09941945e4c0ba70c8ef051266f81 (table test-table [id=5ba00415dbad48caa46f27f913babcf2]) successfully deleted
I20260620 00:06:48.221997 1494 catalog_manager.cc:4841] TS 4742cdc265e14d38abdc9cd9094182f1 (127.0.98.3:42351): tablet c3c09941945e4c0ba70c8ef051266f81 (table test-table [id=5ba00415dbad48caa46f27f913babcf2]) successfully deleted
Jun 20 00:06:48 dist-test-slave-grqt krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914007, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 00:06:48.234161 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:48.225769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52788 (local address 127.0.98.1:35335)
0620 00:06:48.225825 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:48.225827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:48.225933 (+ 106us) server_negotiation.cc:407] Connection header received
0620 00:06:48.226017 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:48.226018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:48.226028 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:48.226076 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:48.226678 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:48.227160 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:48.228029 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:48.228260 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:48.228327 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:48.231282 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:48.231294 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:48.231295 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:48.231303 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:48.232907 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:48.233285 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:48.233289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:48.233289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:48.233354 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:48.233590 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:48.233591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:48.233591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:48.233747 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:48.233908 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:48.233953 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 00:06:48.234042 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260620 00:06:48.234442 1697 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52788
I20260620 00:06:48.241704 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:48.235474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48800 (local address 127.0.98.62:39169)
0620 00:06:48.235521 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:48.235523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:48.235583 (+ 60us) server_negotiation.cc:407] Connection header received
0620 00:06:48.235660 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:48.235660 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:48.235670 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:48.235725 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:48.236241 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:48.236743 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:48.237463 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:48.237677 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:48.237760 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:48.238878 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:48.238879 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:48.238880 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:48.238887 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:48.240238 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:48.240591 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:48.240591 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:48.240592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:48.240649 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:48.240853 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:48.240854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:48.240854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:48.241023 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:48.241149 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:48.241444 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 00:06:48.241578 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 00:06:48.241917 1507 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48800
I20260620 00:06:48.242592 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 1579
I20260620 00:06:48.250111 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 1719
I20260620 00:06:48.256238 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 1859
I20260620 00:06:48.261193 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 1475
2026-06-20T00:06:48Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2623 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:06:48 dist-test-slave-grqt krb5kdc[2075](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:06:48 dist-test-slave-grqt krb5kdc[2075](info): set up 2 sockets
Jun 20 00:06:48 dist-test-slave-grqt krb5kdc[2075](info): commencing operation
krb5kdc: starting...
W20260620 00:06:49.337105 392 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:06:49 dist-test-slave-grqt krb5kdc[2075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:06:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:06:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:06:49.457551 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40575
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:06:49.538213 2088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:06:49.538439 2088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:06:49.538463 2088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:06:49.539883 2088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:06:49.539963 2088 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 00:06:49.539981 2088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:06:49.540076 2088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:06:49.540098 2088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:06:49.540124 2088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:06:49.541953 2088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40575
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:06:49.542258 2088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:06:49.542524 2088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:06:49.548130 2093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:06:49.548143 2094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:06:49.548143 2096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:06:49.548426 2088 server_base.cc:1034] running on GCE node
I20260620 00:06:49.548659 2088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:06:49.549034 2088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:06:49.550196 2088 hybrid_clock.cc:648] HybridClock initialized: now 1781914009550219 us; error 73 us; skew 500 ppm
Jun 20 00:06:49 dist-test-slave-grqt krb5kdc[2075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:06:49.553005 2088 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:06:49.553611 2088 webserver.cc:460] Webserver started at http://127.0.98.62:37825/ using document root <none> and password file <none>
I20260620 00:06:49.553853 2088 fs_manager.cc:359] Metadata directory not provided
I20260620 00:06:49.553929 2088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:06:49.554095 2088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:06:49.555363 2088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "49f4add4939e45c4998254385ad73d44"
format_stamp: "Formatted at 2026-06-20 00:06:49 on dist-test-slave-grqt"
server_key: "125502201abec8dab7efa0e10709eb9e"
server_key_iv: "eebfb08838b82f80b15f4e335354e042"
server_key_version: "encryptionkey@0"
I20260620 00:06:49.555776 2088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "49f4add4939e45c4998254385ad73d44"
format_stamp: "Formatted at 2026-06-20 00:06:49 on dist-test-slave-grqt"
server_key: "125502201abec8dab7efa0e10709eb9e"
server_key_iv: "eebfb08838b82f80b15f4e335354e042"
server_key_version: "encryptionkey@0"
I20260620 00:06:49.557190 2088 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:06:49.558182 2103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:49.558401 2088 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:06:49.558529 2088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "49f4add4939e45c4998254385ad73d44"
format_stamp: "Formatted at 2026-06-20 00:06:49 on dist-test-slave-grqt"
server_key: "125502201abec8dab7efa0e10709eb9e"
server_key_iv: "eebfb08838b82f80b15f4e335354e042"
server_key_version: "encryptionkey@0"
I20260620 00:06:49.558614 2088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:06:49.569737 2088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:06:49.572367 2088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:06:49.572548 2088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:06:49.577657 2088 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40575
I20260620 00:06:49.577713 2165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40575 every 8 connection(s)
I20260620 00:06:49.578074 2088 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:06:49.578770 2166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:06:49.581110 2166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49f4add4939e45c4998254385ad73d44: Bootstrap starting.
I20260620 00:06:49.581804 2166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49f4add4939e45c4998254385ad73d44: Neither blocks nor log segments found. Creating new log.
I20260620 00:06:49.582113 2166 log.cc:824] T 00000000000000000000000000000000 P 49f4add4939e45c4998254385ad73d44: Log is configured to *not* fsync() on all Append() calls
I20260620 00:06:49.582492 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 2088
I20260620 00:06:49.582600 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:06:49.582773 392 external_mini_cluster.cc:1428] Setting key 387f280a3094e2f09dc58acb2d23c1b4
I20260620 00:06:49.582881 2166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49f4add4939e45c4998254385ad73d44: No bootstrap required, opened a new log
I20260620 00:06:49.585816 2166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49f4add4939e45c4998254385ad73d44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f4add4939e45c4998254385ad73d44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } }
I20260620 00:06:49.586005 2166 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49f4add4939e45c4998254385ad73d44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:06:49.586050 2166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49f4add4939e45c4998254385ad73d44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49f4add4939e45c4998254385ad73d44, State: Initialized, Role: FOLLOWER
I20260620 00:06:49.586202 2166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49f4add4939e45c4998254385ad73d44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f4add4939e45c4998254385ad73d44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } }
I20260620 00:06:49.586278 2166 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49f4add4939e45c4998254385ad73d44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:06:49.586308 2166 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49f4add4939e45c4998254385ad73d44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:06:49.586344 2166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49f4add4939e45c4998254385ad73d44 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:06:49.587004 2166 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49f4add4939e45c4998254385ad73d44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f4add4939e45c4998254385ad73d44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } }
I20260620 00:06:49.587102 2166 leader_election.cc:302] T 00000000000000000000000000000000 P 49f4add4939e45c4998254385ad73d44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49f4add4939e45c4998254385ad73d44; no voters:
I20260620 00:06:49.587338 2166 leader_election.cc:288] T 00000000000000000000000000000000 P 49f4add4939e45c4998254385ad73d44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:06:49.587399 2171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49f4add4939e45c4998254385ad73d44 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:06:49.587569 2171 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49f4add4939e45c4998254385ad73d44 [term 1 LEADER]: Becoming Leader. State: Replica: 49f4add4939e45c4998254385ad73d44, State: Running, Role: LEADER
I20260620 00:06:49.587616 2166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49f4add4939e45c4998254385ad73d44 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:06:49.587668 2171 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49f4add4939e45c4998254385ad73d44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f4add4939e45c4998254385ad73d44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } }
I20260620 00:06:49.587999 2171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49f4add4939e45c4998254385ad73d44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49f4add4939e45c4998254385ad73d44. Latest consensus state: current_term: 1 leader_uuid: "49f4add4939e45c4998254385ad73d44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f4add4939e45c4998254385ad73d44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } }
I20260620 00:06:49.588125 2171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49f4add4939e45c4998254385ad73d44 [sys.catalog]: This master's current role is: LEADER
I20260620 00:06:49.588016 2172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49f4add4939e45c4998254385ad73d44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49f4add4939e45c4998254385ad73d44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f4add4939e45c4998254385ad73d44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } }
I20260620 00:06:49.588294 2172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49f4add4939e45c4998254385ad73d44 [sys.catalog]: This master's current role is: LEADER
I20260620 00:06:49.588332 2178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:06:49.588927 2178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:06:49.590792 2178 catalog_manager.cc:1347] Generated new cluster ID: 9da4fe1f54244b5e90dba9f79aee74a5
I20260620 00:06:49.590849 2178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 00:06:49 dist-test-slave-grqt krb5kdc[2075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:06:49.595131 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:49.583336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55514 (local address 127.0.98.62:40575)
0620 00:06:49.583516 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:49.583527 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:49.583578 (+ 51us) server_negotiation.cc:407] Connection header received
0620 00:06:49.584301 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:49.584311 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:49.584379 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:49.584547 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:49.585151 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:49.586519 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:49.587597 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:49.588256 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:49.588353 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:49.591391 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:49.591395 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:49.591395 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:49.591409 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:49.593152 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:49.593615 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:49.593619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:49.593619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:49.593711 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:49.593922 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:49.593923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:49.593923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:49.594166 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:49.594809 (+ 643us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:49.594876 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 00:06:49.594982 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":65,"threads_started":1}
I20260620 00:06:49.602046 2178 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:06:49.602646 2178 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:06:49.606669 2178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49f4add4939e45c4998254385ad73d44: Generated new TSK 0
I20260620 00:06:49.606881 2178 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:06:49.640555 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40575
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:06:49.723557 2192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:06:49.723819 2192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:06:49.723865 2192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:06:49.725107 2192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:06:49.725183 2192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:06:49.725216 2192 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 00:06:49.725248 2192 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 00:06:49.725297 2192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:06:49.726855 2192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:06:49.727147 2192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:06:49.727406 2192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:06:49.732671 2197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:06:49.732770 2192 server_base.cc:1034] running on GCE node
W20260620 00:06:49.732779 2200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:06:49.732671 2198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:06:49.733093 2192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:06:49.733348 2192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:06:49.734583 2192 hybrid_clock.cc:648] HybridClock initialized: now 1781914009734571 us; error 29 us; skew 500 ppm
Jun 20 00:06:49 dist-test-slave-grqt krb5kdc[2075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:06:49.737212 2192 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:06:49.737731 2192 webserver.cc:460] Webserver started at http://127.0.98.1:46213/ using document root <none> and password file <none>
I20260620 00:06:49.737958 2192 fs_manager.cc:359] Metadata directory not provided
I20260620 00:06:49.738004 2192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:06:49.738168 2192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:06:49.739228 2192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-0/data/instance:
uuid: "a2564232d24e4d959fa3c573a3b9deb1"
format_stamp: "Formatted at 2026-06-20 00:06:49 on dist-test-slave-grqt"
server_key: "7055440dbbd459934be31fdd3988b2b6"
server_key_iv: "eda701741f325ad0d2101c24b003a2de"
server_key_version: "encryptionkey@0"
I20260620 00:06:49.739593 2192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a2564232d24e4d959fa3c573a3b9deb1"
format_stamp: "Formatted at 2026-06-20 00:06:49 on dist-test-slave-grqt"
server_key: "7055440dbbd459934be31fdd3988b2b6"
server_key_iv: "eda701741f325ad0d2101c24b003a2de"
server_key_version: "encryptionkey@0"
I20260620 00:06:49.740906 2192 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:06:49.741719 2207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:49.741879 2192 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:49.741987 2192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-0/wal
uuid: "a2564232d24e4d959fa3c573a3b9deb1"
format_stamp: "Formatted at 2026-06-20 00:06:49 on dist-test-slave-grqt"
server_key: "7055440dbbd459934be31fdd3988b2b6"
server_key_iv: "eda701741f325ad0d2101c24b003a2de"
server_key_version: "encryptionkey@0"
I20260620 00:06:49.742087 2192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:06:49.759413 2192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:06:49.761802 2192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:06:49.761970 2192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:06:49.762508 2192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:06:49.762578 2192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:49.762624 2192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:06:49.762672 2192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:49.769351 2192 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37291
I20260620 00:06:49.769461 2322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37291 every 8 connection(s)
I20260620 00:06:49.769754 2192 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 00:06:49 dist-test-slave-grqt krb5kdc[2075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:06:49.773191 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:49.763064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46629 (local address 127.0.98.62:40575)
0620 00:06:49.763122 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:49.763125 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:49.763385 (+ 260us) server_negotiation.cc:407] Connection header received
0620 00:06:49.763883 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:49.763884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:49.763898 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:49.763979 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:49.764992 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:49.765891 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:49.767497 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:49.767760 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:49.767851 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:49.770512 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:49.770513 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:49.770513 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:49.770520 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:49.772011 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:49.772363 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:49.772364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:49.772365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:49.772414 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:49.772674 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:49.772675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:49.772675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:49.772779 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:49.772828 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:49.773022 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 00:06:49.773124 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 00:06:49.773471 2216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:49.763074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.98.1:46629)
0620 00:06:49.763251 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:49.763292 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:49.763784 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:49.764111 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:49.764115 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:49.764538 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:06:49.764872 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:49.764879 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:49.765992 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:49.765993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:49.767403 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:49.767406 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:49.767861 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:49.767862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:49.768010 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:49.768673 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:06:49.768684 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:06:49.770415 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:06:49.772090 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:49.772095 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:49.772097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:49.772299 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:49.772498 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:49.772499 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:49.772499 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:49.772610 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:49.772819 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:06:49.772822 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:06:49.772935 (+ 113us) client_negotiation.cc:769] Sending connection context
0620 00:06:49.773046 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 00:06:49.773175 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":107,"threads_started":1}
I20260620 00:06:49.773999 2323 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40575
I20260620 00:06:49.774088 2323 heartbeater.cc:464] Registering TS with master...
I20260620 00:06:49.774277 2323 heartbeater.cc:511] Master 127.0.98.62:40575 requested a full tablet report, sending...
I20260620 00:06:49.774842 2119 ts_manager.cc:194] Registered new tserver with Master: a2564232d24e4d959fa3c573a3b9deb1 (127.0.98.1:37291)
I20260620 00:06:49.775754 2119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46629
I20260620 00:06:49.776135 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 2192
I20260620 00:06:49.776244 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:06:49.776441 392 external_mini_cluster.cc:1428] Setting key 5a7f6e2791fe73b961c935f713a2989c
I20260620 00:06:49.781421 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:49.775233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58979 (local address 127.0.98.62:40575)
0620 00:06:49.775272 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:49.775274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:49.775396 (+ 122us) server_negotiation.cc:407] Connection header received
0620 00:06:49.775422 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:49.775422 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:49.775432 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:49.775498 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:49.775779 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:49.776453 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:49.777357 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:49.777709 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:49.777783 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:49.778751 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:49.778752 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:49.778753 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:49.778760 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:49.780213 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:49.780600 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:49.780602 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:49.780602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:49.780657 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:49.780874 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:49.780875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:49.780875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:49.780988 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:49.781062 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:49.781230 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 00:06:49.781308 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 00:06:49.781454 2329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:49.775143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.98.1:58979)
0620 00:06:49.775316 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:49.775329 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:49.775387 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:49.775503 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:49.775504 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:49.775658 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:06:49.775711 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:49.775712 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:49.776586 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:49.776587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:49.777296 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:49.777298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:49.777799 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:49.777800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:49.777856 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:49.778341 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:06:49.778347 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:06:49.778651 (+ 304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:06:49.780323 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:49.780324 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:49.780325 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:49.780508 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:49.780732 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:49.780732 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:49.780733 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:49.780793 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:49.781070 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:06:49.781071 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:06:49.781117 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 00:06:49.781202 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 00:06:49.781310 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":109,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:06:49.804720 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40575
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:06:49.886672 2332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:06:49.886888 2332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:06:49.886912 2332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:06:49.888094 2332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:06:49.888170 2332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:06:49.888206 2332 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 00:06:49.888255 2332 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 00:06:49.888289 2332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:06:49.889905 2332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:06:49.890106 2332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:06:49.890286 2332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:06:49.895529 2337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:06:49.895721 2332 server_base.cc:1034] running on GCE node
W20260620 00:06:49.895673 2338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:06:49.895537 2340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:06:49.896070 2332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:06:49.896369 2332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:06:49.897559 2332 hybrid_clock.cc:648] HybridClock initialized: now 1781914009897547 us; error 50 us; skew 500 ppm
Jun 20 00:06:49 dist-test-slave-grqt krb5kdc[2075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:06:49.900534 2332 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:06:49.901163 2332 webserver.cc:460] Webserver started at http://127.0.98.2:42361/ using document root <none> and password file <none>
I20260620 00:06:49.901506 2332 fs_manager.cc:359] Metadata directory not provided
I20260620 00:06:49.901593 2332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:06:49.901746 2332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:06:49.902915 2332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-1/data/instance:
uuid: "84332eccbe6642c886c8ae3489db3b1b"
format_stamp: "Formatted at 2026-06-20 00:06:49 on dist-test-slave-grqt"
server_key: "ede050898ca227f64771eae8a0e0a8df"
server_key_iv: "ccab47b3af21e57193cacaddc8701b62"
server_key_version: "encryptionkey@0"
I20260620 00:06:49.903322 2332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-1/wal/instance:
uuid: "84332eccbe6642c886c8ae3489db3b1b"
format_stamp: "Formatted at 2026-06-20 00:06:49 on dist-test-slave-grqt"
server_key: "ede050898ca227f64771eae8a0e0a8df"
server_key_iv: "ccab47b3af21e57193cacaddc8701b62"
server_key_version: "encryptionkey@0"
I20260620 00:06:49.904917 2332 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:06:49.906038 2347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:49.906404 2332 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:06:49.906558 2332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-1/wal
uuid: "84332eccbe6642c886c8ae3489db3b1b"
format_stamp: "Formatted at 2026-06-20 00:06:49 on dist-test-slave-grqt"
server_key: "ede050898ca227f64771eae8a0e0a8df"
server_key_iv: "ccab47b3af21e57193cacaddc8701b62"
server_key_version: "encryptionkey@0"
I20260620 00:06:49.906654 2332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:06:49.915861 2332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:06:49.918176 2332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:06:49.918351 2332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:06:49.918929 2332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:06:49.918990 2332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:49.919037 2332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:06:49.919057 2332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 00:06:49 dist-test-slave-grqt krb5kdc[2075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:06:49.926393 2332 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36571
I20260620 00:06:49.926813 2332 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:06:49.927270 2462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36571 every 8 connection(s)
I20260620 00:06:49.928814 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:49.919328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41443 (local address 127.0.98.62:40575)
0620 00:06:49.919391 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:49.919393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:49.919703 (+ 310us) server_negotiation.cc:407] Connection header received
0620 00:06:49.920229 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:49.920230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:49.920242 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:49.920299 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:49.921143 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:49.921786 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:49.922844 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:49.923059 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:49.923147 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:49.925986 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:49.925988 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:49.925988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:49.925995 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:49.927679 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:49.928050 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:49.928051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:49.928051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:49.928103 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:49.928302 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:49.928302 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:49.928303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:49.928401 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:49.928447 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:49.928693 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 00:06:49.928752 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 00:06:49.929081 2356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:49.919338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.98.2:41443)
0620 00:06:49.919568 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:49.919609 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:49.920122 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:49.920377 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:49.920381 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:49.920740 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:06:49.921038 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:49.921044 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:49.921888 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:49.921889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:49.922737 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:49.922740 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:49.923157 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:49.923158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:49.923334 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:49.923925 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:06:49.923938 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:06:49.925865 (+ 1927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:06:49.927753 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:49.927757 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:49.927759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:49.927973 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:49.928163 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:49.928164 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:49.928164 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:49.928260 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:49.928451 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:06:49.928455 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:06:49.928596 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 00:06:49.928675 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 00:06:49.928877 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":144,"threads_started":1}
I20260620 00:06:49.929618 2463 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40575
I20260620 00:06:49.929711 2463 heartbeater.cc:464] Registering TS with master...
I20260620 00:06:49.929905 2463 heartbeater.cc:511] Master 127.0.98.62:40575 requested a full tablet report, sending...
I20260620 00:06:49.929910 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 2332
I20260620 00:06:49.930080 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:06:49.930291 2120 ts_manager.cc:194] Registered new tserver with Master: 84332eccbe6642c886c8ae3489db3b1b (127.0.98.2:36571)
I20260620 00:06:49.930455 392 external_mini_cluster.cc:1428] Setting key c7ca7aa3a6880ddc6d5bc0c28aca82f5
I20260620 00:06:49.930923 2120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41443
I20260620 00:06:49.937505 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:49.930829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56051 (local address 127.0.98.62:40575)
0620 00:06:49.930870 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:49.930872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:49.931038 (+ 166us) server_negotiation.cc:407] Connection header received
0620 00:06:49.931060 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:49.931061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:49.931071 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:49.931132 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:49.931690 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:49.932184 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:49.933025 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:49.933390 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:49.933460 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:49.934619 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:49.934620 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:49.934620 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:49.934627 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:49.936202 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:49.936647 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:49.936648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:49.936649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:49.936701 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:49.936928 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:49.936928 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:49.936928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:49.937067 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:49.937129 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:49.937309 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 00:06:49.937425 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 00:06:49.937565 2469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:49.930633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.98.2:56051)
0620 00:06:49.930906 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:49.930917 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:49.930986 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:49.931229 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:49.931231 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:49.931514 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:06:49.931596 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:49.931598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:49.932284 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:49.932285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:49.932940 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:49.932942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:49.933512 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:49.933513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:49.933601 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:49.934036 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:06:49.934042 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:06:49.934510 (+ 468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:06:49.936299 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:49.936300 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:49.936301 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:49.936556 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:49.936787 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:49.936788 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:49.936788 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:49.936861 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:49.937141 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:06:49.937142 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:06:49.937205 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 00:06:49.937280 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 00:06:49.937439 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":104,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:06:49.957973 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40575
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:06:50.041534 2472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:06:50.041785 2472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:06:50.041812 2472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:06:50.042997 2472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:06:50.043033 2472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:06:50.043056 2472 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 00:06:50.043067 2472 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 00:06:50.043084 2472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:06:50.044528 2472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:06:50.044725 2472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:06:50.044906 2472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:06:50.050145 2478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:06:50.050114 2477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:06:50.050120 2480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:06:50.050420 2472 server_base.cc:1034] running on GCE node
I20260620 00:06:50.050657 2472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:06:50.050925 2472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:06:50.052086 2472 hybrid_clock.cc:648] HybridClock initialized: now 1781914010052066 us; error 33 us; skew 500 ppm
Jun 20 00:06:50 dist-test-slave-grqt krb5kdc[2075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:06:50.054770 2472 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:06:50.055243 2472 webserver.cc:460] Webserver started at http://127.0.98.3:42843/ using document root <none> and password file <none>
I20260620 00:06:50.055485 2472 fs_manager.cc:359] Metadata directory not provided
I20260620 00:06:50.055558 2472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:06:50.055707 2472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:06:50.056772 2472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-2/data/instance:
uuid: "66516f726b524477910e30b8165b19f2"
format_stamp: "Formatted at 2026-06-20 00:06:50 on dist-test-slave-grqt"
server_key: "9d2d40b43e51ab1020529dea527848c8"
server_key_iv: "da093f1c9821361fcd25163d0a319f58"
server_key_version: "encryptionkey@0"
I20260620 00:06:50.057130 2472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-2/wal/instance:
uuid: "66516f726b524477910e30b8165b19f2"
format_stamp: "Formatted at 2026-06-20 00:06:50 on dist-test-slave-grqt"
server_key: "9d2d40b43e51ab1020529dea527848c8"
server_key_iv: "da093f1c9821361fcd25163d0a319f58"
server_key_version: "encryptionkey@0"
I20260620 00:06:50.058594 2472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:06:50.059434 2487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:50.059625 2472 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 00:06:50.059738 2472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-2/wal
uuid: "66516f726b524477910e30b8165b19f2"
format_stamp: "Formatted at 2026-06-20 00:06:50 on dist-test-slave-grqt"
server_key: "9d2d40b43e51ab1020529dea527848c8"
server_key_iv: "da093f1c9821361fcd25163d0a319f58"
server_key_version: "encryptionkey@0"
I20260620 00:06:50.059834 2472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:06:50.077112 2472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:06:50.079826 2472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:06:50.080032 2472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:06:50.080631 2472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:06:50.080708 2472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:50.080757 2472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:06:50.080791 2472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:50.088465 2472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38535
I20260620 00:06:50.088577 2602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38535 every 8 connection(s)
Jun 20 00:06:50 dist-test-slave-grqt krb5kdc[2075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:06:50.088999 2472 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:06:50.092399 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:50.081178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49325 (local address 127.0.98.62:40575)
0620 00:06:50.081258 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:50.081261 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:50.081451 (+ 190us) server_negotiation.cc:407] Connection header received
0620 00:06:50.082079 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:50.082081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:50.082092 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:50.082132 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:50.083056 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.084010 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.085766 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.086169 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.086261 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:50.089410 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:50.089411 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:50.089412 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:50.089419 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:50.090777 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:50.091283 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:50.091284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:50.091284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:50.091342 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:50.091766 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:50.091766 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:50.091767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:50.091886 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:50.091947 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:50.092258 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 00:06:50.092332 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 00:06:50.092558 2496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:50.081137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.98.3:49325)
0620 00:06:50.081300 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:50.081339 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:50.081963 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:50.082218 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:50.082228 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:50.082557 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:06:50.082930 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:50.082938 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.084138 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.084139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:50.085657 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:50.085661 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.086274 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.086275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:50.086412 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:50.087055 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:06:50.087067 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:06:50.089262 (+ 2195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:06:50.090885 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:50.090892 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:50.090894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:50.091173 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:50.091451 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:50.091452 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:50.091452 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:50.091676 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:50.091973 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:06:50.091979 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:06:50.092119 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 00:06:50.092219 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 00:06:50.092353 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":96,"threads_started":1}
I20260620 00:06:50.092958 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 2472
I20260620 00:06:50.093055 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:06:50.093128 2603 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40575
I20260620 00:06:50.093210 2603 heartbeater.cc:464] Registering TS with master...
I20260620 00:06:50.093255 392 external_mini_cluster.cc:1428] Setting key b7076a9e147b813a0a78b7c0785262e2
I20260620 00:06:50.093528 2603 heartbeater.cc:511] Master 127.0.98.62:40575 requested a full tablet report, sending...
I20260620 00:06:50.093886 2120 ts_manager.cc:194] Registered new tserver with Master: 66516f726b524477910e30b8165b19f2 (127.0.98.3:38535)
I20260620 00:06:50.094372 2120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49325
I20260620 00:06:50.094981 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:06:50.101513 2609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:50.094266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.98.3:36757)
0620 00:06:50.094397 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:50.094407 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:50.094472 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:50.094586 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:50.094586 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:50.094785 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:06:50.094882 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:50.094885 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.095730 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.095731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:50.096617 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:50.096619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.097147 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.097148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:50.097212 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:50.097792 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:06:50.097798 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:06:50.098124 (+ 326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:06:50.100020 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:50.100021 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:50.100022 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:50.100263 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:50.100515 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:50.100516 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:50.100516 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:50.100594 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:50.100898 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:06:50.100899 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:06:50.100952 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 00:06:50.101120 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 00:06:50.101295 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":8,"thread_start_us":131,"threads_started":1}
I20260620 00:06:50.101537 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:50.094336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36757 (local address 127.0.98.62:40575)
0620 00:06:50.094385 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:50.094387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:50.094457 (+ 70us) server_negotiation.cc:407] Connection header received
0620 00:06:50.094522 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:50.094523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:50.094533 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:50.094584 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:50.094946 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.095631 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.096724 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.097045 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.097130 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:50.098217 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:50.098218 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:50.098218 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:50.098226 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:50.099897 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:50.100366 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:50.100367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:50.100367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:50.100423 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:50.100703 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:50.100704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:50.100705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:50.100822 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:50.100889 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:50.101295 (+ 406us) server_negotiation.cc:299] Negotiation successful
0620 00:06:50.101412 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
Jun 20 00:06:50 dist-test-slave-grqt krb5kdc[2075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 00:06:50 dist-test-slave-grqt krb5kdc[2075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:06:50.119457 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:50.112126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55528 (local address 127.0.98.62:40575)
0620 00:06:50.112196 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:50.112198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:50.112208 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:06:50.112237 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:50.112238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:50.112252 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:50.112319 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:50.112890 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.113572 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.114315 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.114519 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.114608 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:50.116934 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:50.116936 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:50.116936 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:50.116944 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:50.118312 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:50.118680 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:50.118681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:50.118682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:50.118742 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:50.118937 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:50.118938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:50.118939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:50.119050 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:50.119162 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:50.119274 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 00:06:50.119352 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 00:06:50.121210 2120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55528:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:06:50.122994 2120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:06:50.129956 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:50.125581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38535 (local address 127.0.0.1:49828)
0620 00:06:50.125907 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:50.125917 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:50.125961 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:50.126276 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:50.126277 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:50.126278 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:06:50.126315 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:50.126317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.127527 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.127527 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:50.128687 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:50.128689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.129704 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.129705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:50.129772 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:50.129776 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:06:50.129835 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 00:06:50.129874 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":45,"thread_start_us":78,"threads_started":1}
I20260620 00:06:50.129978 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:50.125754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49828 (local address 127.0.98.3:38535)
0620 00:06:50.126106 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:50.126111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:50.126121 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:06:50.126160 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:50.126162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:50.126177 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:50.126380 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:06:50.126437 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.127454 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.128772 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.129606 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.129701 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:50.129756 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:50.129850 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 00:06:50.129882 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":84,"threads_started":1}
I20260620 00:06:50.130438 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:50.125443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39774 (local address 127.0.98.2:36571)
0620 00:06:50.125625 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:50.125636 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:50.125645 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:06:50.125701 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:50.125703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:50.125721 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:50.125809 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:06:50.126056 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.127280 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.128596 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.129329 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.129436 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:50.129501 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:50.129902 (+ 401us) server_negotiation.cc:299] Negotiation successful
0620 00:06:50.129944 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":103,"threads_started":1}
I20260620 00:06:50.130643 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:50.125312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36571 (local address 127.0.0.1:39774)
0620 00:06:50.125446 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:50.125485 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:50.125549 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:50.125889 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:50.125892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:50.125895 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:06:50.125971 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:50.125972 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.127372 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.127373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:50.128491 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:50.128494 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.129453 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.129454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:50.129600 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:50.129621 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 00:06:50.129701 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 00:06:50.129756 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":33,"thread_start_us":105,"threads_started":1}
I20260620 00:06:50.130885 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:50.125752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37291 (local address 127.0.0.1:38754)
0620 00:06:50.126077 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:50.126086 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:50.126167 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:50.126448 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:50.126449 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:50.126451 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:06:50.126495 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:50.126496 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.127379 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.127380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:50.128687 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:50.128689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.130606 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.130607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:50.130673 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:50.130677 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:06:50.130734 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 00:06:50.130820 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":48,"thread_start_us":72,"threads_started":1}
I20260620 00:06:50.130834 2537 tablet_service.cc:1467] Processing CreateTablet for tablet 5ca4f99f419f4607915f719140fa623e (DEFAULT_TABLE table=test-table [id=12fec3a50134400b959f307a7f1b799f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:06:50.131179 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:50.125864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38754 (local address 127.0.98.1:37291)
0620 00:06:50.126023 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:50.126028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:50.126157 (+ 129us) server_negotiation.cc:407] Connection header received
0620 00:06:50.126294 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:50.126296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:50.126311 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:50.126384 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:06:50.126584 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.127276 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.129483 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.130415 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.130486 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:50.130548 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:50.131045 (+ 497us) server_negotiation.cc:299] Negotiation successful
0620 00:06:50.131087 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":97,"threads_started":1}
I20260620 00:06:50.131266 2537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ca4f99f419f4607915f719140fa623e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:06:50.131369 2397 tablet_service.cc:1467] Processing CreateTablet for tablet 5ca4f99f419f4607915f719140fa623e (DEFAULT_TABLE table=test-table [id=12fec3a50134400b959f307a7f1b799f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:06:50.131703 2397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ca4f99f419f4607915f719140fa623e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:06:50.131734 2257 tablet_service.cc:1467] Processing CreateTablet for tablet 5ca4f99f419f4607915f719140fa623e (DEFAULT_TABLE table=test-table [id=12fec3a50134400b959f307a7f1b799f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:06:50.132033 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ca4f99f419f4607915f719140fa623e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:06:50.134133 2626 tablet_bootstrap.cc:492] T 5ca4f99f419f4607915f719140fa623e P 66516f726b524477910e30b8165b19f2: Bootstrap starting.
I20260620 00:06:50.135042 2626 tablet_bootstrap.cc:654] T 5ca4f99f419f4607915f719140fa623e P 66516f726b524477910e30b8165b19f2: Neither blocks nor log segments found. Creating new log.
I20260620 00:06:50.135413 2626 log.cc:824] T 5ca4f99f419f4607915f719140fa623e P 66516f726b524477910e30b8165b19f2: Log is configured to *not* fsync() on all Append() calls
I20260620 00:06:50.135429 2627 tablet_bootstrap.cc:492] T 5ca4f99f419f4607915f719140fa623e P 84332eccbe6642c886c8ae3489db3b1b: Bootstrap starting.
I20260620 00:06:50.136081 2627 tablet_bootstrap.cc:654] T 5ca4f99f419f4607915f719140fa623e P 84332eccbe6642c886c8ae3489db3b1b: Neither blocks nor log segments found. Creating new log.
I20260620 00:06:50.136159 2626 tablet_bootstrap.cc:492] T 5ca4f99f419f4607915f719140fa623e P 66516f726b524477910e30b8165b19f2: No bootstrap required, opened a new log
I20260620 00:06:50.136225 2626 ts_tablet_manager.cc:1397] T 5ca4f99f419f4607915f719140fa623e P 66516f726b524477910e30b8165b19f2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:06:50.136410 2627 log.cc:824] T 5ca4f99f419f4607915f719140fa623e P 84332eccbe6642c886c8ae3489db3b1b: Log is configured to *not* fsync() on all Append() calls
I20260620 00:06:50.136739 2628 tablet_bootstrap.cc:492] T 5ca4f99f419f4607915f719140fa623e P a2564232d24e4d959fa3c573a3b9deb1: Bootstrap starting.
I20260620 00:06:50.137107 2627 tablet_bootstrap.cc:492] T 5ca4f99f419f4607915f719140fa623e P 84332eccbe6642c886c8ae3489db3b1b: No bootstrap required, opened a new log
I20260620 00:06:50.137190 2627 ts_tablet_manager.cc:1397] T 5ca4f99f419f4607915f719140fa623e P 84332eccbe6642c886c8ae3489db3b1b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:06:50.137341 2628 tablet_bootstrap.cc:654] T 5ca4f99f419f4607915f719140fa623e P a2564232d24e4d959fa3c573a3b9deb1: Neither blocks nor log segments found. Creating new log.
I20260620 00:06:50.137714 2628 log.cc:824] T 5ca4f99f419f4607915f719140fa623e P a2564232d24e4d959fa3c573a3b9deb1: Log is configured to *not* fsync() on all Append() calls
I20260620 00:06:50.138469 2628 tablet_bootstrap.cc:492] T 5ca4f99f419f4607915f719140fa623e P a2564232d24e4d959fa3c573a3b9deb1: No bootstrap required, opened a new log
I20260620 00:06:50.138538 2628 ts_tablet_manager.cc:1397] T 5ca4f99f419f4607915f719140fa623e P a2564232d24e4d959fa3c573a3b9deb1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:06:50.138690 2626 raft_consensus.cc:348] T 5ca4f99f419f4607915f719140fa623e P 66516f726b524477910e30b8165b19f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } } peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } }
I20260620 00:06:50.138855 2626 raft_consensus.cc:374] T 5ca4f99f419f4607915f719140fa623e P 66516f726b524477910e30b8165b19f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:06:50.138898 2626 raft_consensus.cc:729] T 5ca4f99f419f4607915f719140fa623e P 66516f726b524477910e30b8165b19f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66516f726b524477910e30b8165b19f2, State: Initialized, Role: FOLLOWER
I20260620 00:06:50.139034 2626 consensus_queue.cc:260] T 5ca4f99f419f4607915f719140fa623e P 66516f726b524477910e30b8165b19f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } } peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } }
I20260620 00:06:50.139071 2627 raft_consensus.cc:348] T 5ca4f99f419f4607915f719140fa623e P 84332eccbe6642c886c8ae3489db3b1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } } peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } }
I20260620 00:06:50.139243 2627 raft_consensus.cc:374] T 5ca4f99f419f4607915f719140fa623e P 84332eccbe6642c886c8ae3489db3b1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:06:50.139286 2626 ts_tablet_manager.cc:1428] T 5ca4f99f419f4607915f719140fa623e P 66516f726b524477910e30b8165b19f2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:06:50.139288 2627 raft_consensus.cc:729] T 5ca4f99f419f4607915f719140fa623e P 84332eccbe6642c886c8ae3489db3b1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84332eccbe6642c886c8ae3489db3b1b, State: Initialized, Role: FOLLOWER
I20260620 00:06:50.139412 2603 heartbeater.cc:503] Master 127.0.98.62:40575 was elected leader, sending a full tablet report...
I20260620 00:06:50.139406 2627 consensus_queue.cc:260] T 5ca4f99f419f4607915f719140fa623e P 84332eccbe6642c886c8ae3489db3b1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } } peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } }
I20260620 00:06:50.139654 2627 ts_tablet_manager.cc:1428] T 5ca4f99f419f4607915f719140fa623e P 84332eccbe6642c886c8ae3489db3b1b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 00:06:50.139698 2463 heartbeater.cc:503] Master 127.0.98.62:40575 was elected leader, sending a full tablet report...
I20260620 00:06:50.140733 2628 raft_consensus.cc:348] T 5ca4f99f419f4607915f719140fa623e P a2564232d24e4d959fa3c573a3b9deb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } } peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } }
I20260620 00:06:50.140854 2628 raft_consensus.cc:374] T 5ca4f99f419f4607915f719140fa623e P a2564232d24e4d959fa3c573a3b9deb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:06:50.140875 2628 raft_consensus.cc:729] T 5ca4f99f419f4607915f719140fa623e P a2564232d24e4d959fa3c573a3b9deb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2564232d24e4d959fa3c573a3b9deb1, State: Initialized, Role: FOLLOWER
I20260620 00:06:50.140995 2628 consensus_queue.cc:260] T 5ca4f99f419f4607915f719140fa623e P a2564232d24e4d959fa3c573a3b9deb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } } peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } }
I20260620 00:06:50.141242 2628 ts_tablet_manager.cc:1428] T 5ca4f99f419f4607915f719140fa623e P a2564232d24e4d959fa3c573a3b9deb1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:06:50.141297 2323 heartbeater.cc:503] Master 127.0.98.62:40575 was elected leader, sending a full tablet report...
I20260620 00:06:50.168778 2633 raft_consensus.cc:482] T 5ca4f99f419f4607915f719140fa623e P 84332eccbe6642c886c8ae3489db3b1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:06:50.168901 2633 raft_consensus.cc:504] T 5ca4f99f419f4607915f719140fa623e P 84332eccbe6642c886c8ae3489db3b1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } } peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } }
I20260620 00:06:50.169399 2633 leader_election.cc:288] T 5ca4f99f419f4607915f719140fa623e P 84332eccbe6642c886c8ae3489db3b1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66516f726b524477910e30b8165b19f2 (127.0.98.3:38535), a2564232d24e4d959fa3c573a3b9deb1 (127.0.98.1:37291)
I20260620 00:06:50.173182 2356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:50.169628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38535 (local address 127.0.98.2:46205)
0620 00:06:50.169674 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:50.169714 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:50.169779 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:50.169975 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:50.169976 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:50.169978 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:06:50.170053 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:50.170055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.170936 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.170937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:50.172149 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:50.172152 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.172895 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.172897 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:50.172983 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:50.172988 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:06:50.173057 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 00:06:50.173104 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260620 00:06:50.173207 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:50.169759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46205 (local address 127.0.98.3:38535)
0620 00:06:50.169806 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:50.169808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:50.169819 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:06:50.169849 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:50.169850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:50.169862 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:50.169921 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:06:50.170127 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.170855 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.172285 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.172829 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.172895 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:50.172905 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:50.173075 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 00:06:50.173117 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 00:06:50.173784 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:50.169972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53859 (local address 127.0.98.1:37291)
0620 00:06:50.170198 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:50.170199 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:50.170207 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:06:50.170258 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:50.170258 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:50.170269 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:50.170343 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:06:50.170596 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.171345 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.172575 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.173129 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.173172 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:50.173181 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:50.173657 (+ 476us) server_negotiation.cc:299] Negotiation successful
0620 00:06:50.173712 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201}
I20260620 00:06:50.173794 2557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca4f99f419f4607915f719140fa623e" candidate_uuid: "84332eccbe6642c886c8ae3489db3b1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66516f726b524477910e30b8165b19f2" is_pre_election: true
I20260620 00:06:50.173966 2557 raft_consensus.cc:2454] T 5ca4f99f419f4607915f719140fa623e P 66516f726b524477910e30b8165b19f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84332eccbe6642c886c8ae3489db3b1b in term 0.
I20260620 00:06:50.174013 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:50.169741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37291 (local address 127.0.98.2:53859)
0620 00:06:50.169915 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:50.169925 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:50.169966 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:50.170384 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:50.170385 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:50.170387 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:06:50.170475 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:50.170476 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.171440 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.171441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:50.172488 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:50.172489 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.173465 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.173466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:50.173568 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:50.173572 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:06:50.173881 (+ 309us) client_negotiation.cc:240] Negotiation successful
0620 00:06:50.173933 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":123,"threads_started":1}
I20260620 00:06:50.174492 2349 leader_election.cc:302] T 5ca4f99f419f4607915f719140fa623e P 84332eccbe6642c886c8ae3489db3b1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66516f726b524477910e30b8165b19f2, 84332eccbe6642c886c8ae3489db3b1b; no voters:
I20260620 00:06:50.174710 2633 raft_consensus.cc:2781] T 5ca4f99f419f4607915f719140fa623e P 84332eccbe6642c886c8ae3489db3b1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:06:50.174769 2633 raft_consensus.cc:482] T 5ca4f99f419f4607915f719140fa623e P 84332eccbe6642c886c8ae3489db3b1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:06:50.174801 2633 raft_consensus.cc:3037] T 5ca4f99f419f4607915f719140fa623e P 84332eccbe6642c886c8ae3489db3b1b [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:06:50.174867 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca4f99f419f4607915f719140fa623e" candidate_uuid: "84332eccbe6642c886c8ae3489db3b1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2564232d24e4d959fa3c573a3b9deb1" is_pre_election: true
I20260620 00:06:50.175024 2277 raft_consensus.cc:2454] T 5ca4f99f419f4607915f719140fa623e P a2564232d24e4d959fa3c573a3b9deb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84332eccbe6642c886c8ae3489db3b1b in term 0.
I20260620 00:06:50.175628 2633 raft_consensus.cc:504] T 5ca4f99f419f4607915f719140fa623e P 84332eccbe6642c886c8ae3489db3b1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } } peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } }
I20260620 00:06:50.175774 2633 leader_election.cc:288] T 5ca4f99f419f4607915f719140fa623e P 84332eccbe6642c886c8ae3489db3b1b [CANDIDATE]: Term 1 election: Requested vote from peers 66516f726b524477910e30b8165b19f2 (127.0.98.3:38535), a2564232d24e4d959fa3c573a3b9deb1 (127.0.98.1:37291)
I20260620 00:06:50.175940 2557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca4f99f419f4607915f719140fa623e" candidate_uuid: "84332eccbe6642c886c8ae3489db3b1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66516f726b524477910e30b8165b19f2"
I20260620 00:06:50.175949 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca4f99f419f4607915f719140fa623e" candidate_uuid: "84332eccbe6642c886c8ae3489db3b1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2564232d24e4d959fa3c573a3b9deb1"
I20260620 00:06:50.176033 2557 raft_consensus.cc:3037] T 5ca4f99f419f4607915f719140fa623e P 66516f726b524477910e30b8165b19f2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:06:50.176105 2277 raft_consensus.cc:3037] T 5ca4f99f419f4607915f719140fa623e P a2564232d24e4d959fa3c573a3b9deb1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:06:50.176798 2557 raft_consensus.cc:2454] T 5ca4f99f419f4607915f719140fa623e P 66516f726b524477910e30b8165b19f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84332eccbe6642c886c8ae3489db3b1b in term 1.
I20260620 00:06:50.176803 2277 raft_consensus.cc:2454] T 5ca4f99f419f4607915f719140fa623e P a2564232d24e4d959fa3c573a3b9deb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84332eccbe6642c886c8ae3489db3b1b in term 1.
I20260620 00:06:50.176960 2349 leader_election.cc:302] T 5ca4f99f419f4607915f719140fa623e P 84332eccbe6642c886c8ae3489db3b1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84332eccbe6642c886c8ae3489db3b1b, a2564232d24e4d959fa3c573a3b9deb1; no voters:
I20260620 00:06:50.177098 2633 raft_consensus.cc:2781] T 5ca4f99f419f4607915f719140fa623e P 84332eccbe6642c886c8ae3489db3b1b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:06:50.177255 2633 raft_consensus.cc:686] T 5ca4f99f419f4607915f719140fa623e P 84332eccbe6642c886c8ae3489db3b1b [term 1 LEADER]: Becoming Leader. State: Replica: 84332eccbe6642c886c8ae3489db3b1b, State: Running, Role: LEADER
I20260620 00:06:50.177503 2633 consensus_queue.cc:237] T 5ca4f99f419f4607915f719140fa623e P 84332eccbe6642c886c8ae3489db3b1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } } peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } }
W20260620 00:06:50.178232 2464 tablet.cc:2369] T 5ca4f99f419f4607915f719140fa623e P 84332eccbe6642c886c8ae3489db3b1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:06:50.178992 2120 catalog_manager.cc:5495] T 5ca4f99f419f4607915f719140fa623e P 84332eccbe6642c886c8ae3489db3b1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 84332eccbe6642c886c8ae3489db3b1b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "84332eccbe6642c886c8ae3489db3b1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } health_report { overall_health: UNKNOWN } } }
Jun 20 00:06:50 dist-test-slave-grqt krb5kdc[2075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:06:50.213950 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:50.206308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.0.1:55540)
0620 00:06:50.206501 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:50.206512 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:50.206580 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:50.206728 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:50.206729 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:50.207045 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:06:50.207120 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:50.207123 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.207880 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.207881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:50.208428 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:50.208430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.208795 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.208796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:50.208841 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:50.209468 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:06:50.209472 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:06:50.211068 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:06:50.212806 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:50.212812 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:50.212814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:50.213014 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:50.213229 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:50.213230 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:50.213230 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:50.213308 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:50.213666 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:06:50.213672 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:06:50.213720 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 00:06:50.213778 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 00:06:50.213875 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":90,"threads_started":1}
I20260620 00:06:50.214061 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:50.206368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55540 (local address 127.0.98.62:40575)
0620 00:06:50.206441 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:50.206443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:50.206586 (+ 143us) server_negotiation.cc:407] Connection header received
0620 00:06:50.206620 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:50.206621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:50.206630 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:50.206690 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:50.207212 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.207774 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.208511 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.208719 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.208781 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:50.211219 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:50.211220 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:50.211220 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:50.211227 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:50.212726 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:50.213086 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:50.213086 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:50.213087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:50.213140 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:50.213417 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:50.213418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:50.213419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:50.213601 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:50.213673 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:50.213812 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 00:06:50.213938 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":37}
I20260620 00:06:50.216288 2120 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55540:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 00:06:50.216496 2120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:06:50.218067 2397 tablet_service.cc:1467] Processing CreateTablet for tablet 59a42d91008d4dffaedbc0a9a2673e99 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ba52b89ffef6469f9c48a942a803dfa8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 00:06:50.218058 2537 tablet_service.cc:1467] Processing CreateTablet for tablet 59a42d91008d4dffaedbc0a9a2673e99 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ba52b89ffef6469f9c48a942a803dfa8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 00:06:50.218057 2257 tablet_service.cc:1467] Processing CreateTablet for tablet 59a42d91008d4dffaedbc0a9a2673e99 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ba52b89ffef6469f9c48a942a803dfa8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 00:06:50.218251 2537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59a42d91008d4dffaedbc0a9a2673e99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:06:50.218251 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59a42d91008d4dffaedbc0a9a2673e99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:06:50.218253 2397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59a42d91008d4dffaedbc0a9a2673e99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:06:50.219545 2627 tablet_bootstrap.cc:492] T 59a42d91008d4dffaedbc0a9a2673e99 P 84332eccbe6642c886c8ae3489db3b1b: Bootstrap starting.
I20260620 00:06:50.219748 2626 tablet_bootstrap.cc:492] T 59a42d91008d4dffaedbc0a9a2673e99 P 66516f726b524477910e30b8165b19f2: Bootstrap starting.
I20260620 00:06:50.220041 2627 tablet_bootstrap.cc:654] T 59a42d91008d4dffaedbc0a9a2673e99 P 84332eccbe6642c886c8ae3489db3b1b: Neither blocks nor log segments found. Creating new log.
I20260620 00:06:50.220357 2626 tablet_bootstrap.cc:654] T 59a42d91008d4dffaedbc0a9a2673e99 P 66516f726b524477910e30b8165b19f2: Neither blocks nor log segments found. Creating new log.
I20260620 00:06:50.220796 2628 tablet_bootstrap.cc:492] T 59a42d91008d4dffaedbc0a9a2673e99 P a2564232d24e4d959fa3c573a3b9deb1: Bootstrap starting.
I20260620 00:06:50.220853 2627 tablet_bootstrap.cc:492] T 59a42d91008d4dffaedbc0a9a2673e99 P 84332eccbe6642c886c8ae3489db3b1b: No bootstrap required, opened a new log
I20260620 00:06:50.220893 2627 ts_tablet_manager.cc:1397] T 59a42d91008d4dffaedbc0a9a2673e99 P 84332eccbe6642c886c8ae3489db3b1b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:06:50.221030 2626 tablet_bootstrap.cc:492] T 59a42d91008d4dffaedbc0a9a2673e99 P 66516f726b524477910e30b8165b19f2: No bootstrap required, opened a new log
I20260620 00:06:50.221065 2626 ts_tablet_manager.cc:1397] T 59a42d91008d4dffaedbc0a9a2673e99 P 66516f726b524477910e30b8165b19f2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:06:50.221086 2627 raft_consensus.cc:348] T 59a42d91008d4dffaedbc0a9a2673e99 P 84332eccbe6642c886c8ae3489db3b1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } }
I20260620 00:06:50.221199 2627 raft_consensus.cc:374] T 59a42d91008d4dffaedbc0a9a2673e99 P 84332eccbe6642c886c8ae3489db3b1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:06:50.221227 2627 raft_consensus.cc:729] T 59a42d91008d4dffaedbc0a9a2673e99 P 84332eccbe6642c886c8ae3489db3b1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84332eccbe6642c886c8ae3489db3b1b, State: Initialized, Role: FOLLOWER
I20260620 00:06:50.221206 2626 raft_consensus.cc:348] T 59a42d91008d4dffaedbc0a9a2673e99 P 66516f726b524477910e30b8165b19f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } }
I20260620 00:06:50.221254 2626 raft_consensus.cc:374] T 59a42d91008d4dffaedbc0a9a2673e99 P 66516f726b524477910e30b8165b19f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:06:50.221268 2626 raft_consensus.cc:729] T 59a42d91008d4dffaedbc0a9a2673e99 P 66516f726b524477910e30b8165b19f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66516f726b524477910e30b8165b19f2, State: Initialized, Role: FOLLOWER
I20260620 00:06:50.221285 2627 consensus_queue.cc:260] T 59a42d91008d4dffaedbc0a9a2673e99 P 84332eccbe6642c886c8ae3489db3b1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } }
I20260620 00:06:50.221330 2626 consensus_queue.cc:260] T 59a42d91008d4dffaedbc0a9a2673e99 P 66516f726b524477910e30b8165b19f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } }
I20260620 00:06:50.221411 2627 ts_tablet_manager.cc:1428] T 59a42d91008d4dffaedbc0a9a2673e99 P 84332eccbe6642c886c8ae3489db3b1b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 00:06:50.221450 2626 ts_tablet_manager.cc:1428] T 59a42d91008d4dffaedbc0a9a2673e99 P 66516f726b524477910e30b8165b19f2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 00:06:50.221589 2628 tablet_bootstrap.cc:654] T 59a42d91008d4dffaedbc0a9a2673e99 P a2564232d24e4d959fa3c573a3b9deb1: Neither blocks nor log segments found. Creating new log.
I20260620 00:06:50.221587 2632 tablet_replica.cc:439] T 59a42d91008d4dffaedbc0a9a2673e99 P 66516f726b524477910e30b8165b19f2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } } }
I20260620 00:06:50.221683 2632 tablet_replica.cc:442] T 59a42d91008d4dffaedbc0a9a2673e99 P 66516f726b524477910e30b8165b19f2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:06:50.222043 2633 tablet_replica.cc:439] T 59a42d91008d4dffaedbc0a9a2673e99 P 84332eccbe6642c886c8ae3489db3b1b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } } }
I20260620 00:06:50.222150 2633 tablet_replica.cc:442] T 59a42d91008d4dffaedbc0a9a2673e99 P 84332eccbe6642c886c8ae3489db3b1b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:06:50.222455 2628 tablet_bootstrap.cc:492] T 59a42d91008d4dffaedbc0a9a2673e99 P a2564232d24e4d959fa3c573a3b9deb1: No bootstrap required, opened a new log
I20260620 00:06:50.222519 2628 ts_tablet_manager.cc:1397] T 59a42d91008d4dffaedbc0a9a2673e99 P a2564232d24e4d959fa3c573a3b9deb1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:06:50.222698 2628 raft_consensus.cc:348] T 59a42d91008d4dffaedbc0a9a2673e99 P a2564232d24e4d959fa3c573a3b9deb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } }
I20260620 00:06:50.222769 2628 raft_consensus.cc:374] T 59a42d91008d4dffaedbc0a9a2673e99 P a2564232d24e4d959fa3c573a3b9deb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:06:50.222786 2628 raft_consensus.cc:729] T 59a42d91008d4dffaedbc0a9a2673e99 P a2564232d24e4d959fa3c573a3b9deb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2564232d24e4d959fa3c573a3b9deb1, State: Initialized, Role: FOLLOWER
I20260620 00:06:50.222853 2628 consensus_queue.cc:260] T 59a42d91008d4dffaedbc0a9a2673e99 P a2564232d24e4d959fa3c573a3b9deb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } }
I20260620 00:06:50.222960 2628 ts_tablet_manager.cc:1428] T 59a42d91008d4dffaedbc0a9a2673e99 P a2564232d24e4d959fa3c573a3b9deb1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:50.223035 2634 tablet_replica.cc:439] T 59a42d91008d4dffaedbc0a9a2673e99 P a2564232d24e4d959fa3c573a3b9deb1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } } }
I20260620 00:06:50.223119 2634 tablet_replica.cc:442] T 59a42d91008d4dffaedbc0a9a2673e99 P a2564232d24e4d959fa3c573a3b9deb1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:06:50.264459 2634 raft_consensus.cc:482] T 59a42d91008d4dffaedbc0a9a2673e99 P a2564232d24e4d959fa3c573a3b9deb1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:06:50.264597 2634 raft_consensus.cc:504] T 59a42d91008d4dffaedbc0a9a2673e99 P a2564232d24e4d959fa3c573a3b9deb1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } }
I20260620 00:06:50.264974 2634 leader_election.cc:288] T 59a42d91008d4dffaedbc0a9a2673e99 P a2564232d24e4d959fa3c573a3b9deb1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84332eccbe6642c886c8ae3489db3b1b (127.0.98.2:36571), 66516f726b524477910e30b8165b19f2 (127.0.98.3:38535)
I20260620 00:06:50.267912 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:50.265232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37643 (local address 127.0.98.2:36571)
0620 00:06:50.265278 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:50.265280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:50.265289 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:06:50.265325 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:50.265326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:50.265337 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:50.265433 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:06:50.265659 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.266219 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.267166 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.267538 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.267625 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:50.267635 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:50.267797 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 00:06:50.267836 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 00:06:50.267925 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:50.265424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46603 (local address 127.0.98.3:38535)
0620 00:06:50.265481 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:50.265482 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:50.265490 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:06:50.265516 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:50.265517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:50.265532 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:50.265599 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:06:50.265758 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.266299 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.267206 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.267561 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.267640 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:50.267650 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:50.267814 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 00:06:50.267852 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 00:06:50.267962 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:50.265255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38535 (local address 127.0.98.1:46603)
0620 00:06:50.265396 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:50.265405 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:50.265447 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:50.265602 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:50.265603 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:50.265605 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:06:50.265683 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:50.265685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.266387 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.266388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:50.267111 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:50.267113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.267668 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.267669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:50.267731 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:50.267734 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:06:50.267795 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 00:06:50.267851 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":60,"threads_started":1}
I20260620 00:06:50.267968 2216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:50.265157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36571 (local address 127.0.98.1:37643)
0620 00:06:50.265215 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:50.265227 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:50.265274 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:50.265506 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:50.265507 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:50.265509 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:06:50.265578 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:50.265579 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.266310 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.266311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:50.267069 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:50.267071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.267635 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.267636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:50.267709 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:50.267713 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:06:50.267787 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 00:06:50.267831 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":1}
I20260620 00:06:50.268301 2557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59a42d91008d4dffaedbc0a9a2673e99" candidate_uuid: "a2564232d24e4d959fa3c573a3b9deb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66516f726b524477910e30b8165b19f2" is_pre_election: true
I20260620 00:06:50.268342 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59a42d91008d4dffaedbc0a9a2673e99" candidate_uuid: "a2564232d24e4d959fa3c573a3b9deb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84332eccbe6642c886c8ae3489db3b1b" is_pre_election: true
I20260620 00:06:50.268404 2557 raft_consensus.cc:2454] T 59a42d91008d4dffaedbc0a9a2673e99 P 66516f726b524477910e30b8165b19f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2564232d24e4d959fa3c573a3b9deb1 in term 0.
I20260620 00:06:50.268435 2417 raft_consensus.cc:2454] T 59a42d91008d4dffaedbc0a9a2673e99 P 84332eccbe6642c886c8ae3489db3b1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2564232d24e4d959fa3c573a3b9deb1 in term 0.
I20260620 00:06:50.268628 2209 leader_election.cc:302] T 59a42d91008d4dffaedbc0a9a2673e99 P a2564232d24e4d959fa3c573a3b9deb1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66516f726b524477910e30b8165b19f2, a2564232d24e4d959fa3c573a3b9deb1; no voters:
I20260620 00:06:50.268800 2634 raft_consensus.cc:2781] T 59a42d91008d4dffaedbc0a9a2673e99 P a2564232d24e4d959fa3c573a3b9deb1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:06:50.268846 2634 raft_consensus.cc:482] T 59a42d91008d4dffaedbc0a9a2673e99 P a2564232d24e4d959fa3c573a3b9deb1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:06:50.268867 2634 raft_consensus.cc:3037] T 59a42d91008d4dffaedbc0a9a2673e99 P a2564232d24e4d959fa3c573a3b9deb1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:06:50.269629 2634 raft_consensus.cc:504] T 59a42d91008d4dffaedbc0a9a2673e99 P a2564232d24e4d959fa3c573a3b9deb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } }
I20260620 00:06:50.269765 2634 leader_election.cc:288] T 59a42d91008d4dffaedbc0a9a2673e99 P a2564232d24e4d959fa3c573a3b9deb1 [CANDIDATE]: Term 1 election: Requested vote from peers 84332eccbe6642c886c8ae3489db3b1b (127.0.98.2:36571), 66516f726b524477910e30b8165b19f2 (127.0.98.3:38535)
I20260620 00:06:50.269985 2557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59a42d91008d4dffaedbc0a9a2673e99" candidate_uuid: "a2564232d24e4d959fa3c573a3b9deb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66516f726b524477910e30b8165b19f2"
I20260620 00:06:50.270056 2557 raft_consensus.cc:3037] T 59a42d91008d4dffaedbc0a9a2673e99 P 66516f726b524477910e30b8165b19f2 [term 0 FOLLOWER]: Advancing to term 1
W20260620 00:06:50.270464 2324 tablet.cc:2369] T 59a42d91008d4dffaedbc0a9a2673e99 P a2564232d24e4d959fa3c573a3b9deb1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:06:50.270632 2557 raft_consensus.cc:2454] T 59a42d91008d4dffaedbc0a9a2673e99 P 66516f726b524477910e30b8165b19f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2564232d24e4d959fa3c573a3b9deb1 in term 1.
I20260620 00:06:50.270758 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59a42d91008d4dffaedbc0a9a2673e99" candidate_uuid: "a2564232d24e4d959fa3c573a3b9deb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84332eccbe6642c886c8ae3489db3b1b"
I20260620 00:06:50.270812 2209 leader_election.cc:302] T 59a42d91008d4dffaedbc0a9a2673e99 P a2564232d24e4d959fa3c573a3b9deb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66516f726b524477910e30b8165b19f2, a2564232d24e4d959fa3c573a3b9deb1; no voters:
I20260620 00:06:50.270838 2417 raft_consensus.cc:3037] T 59a42d91008d4dffaedbc0a9a2673e99 P 84332eccbe6642c886c8ae3489db3b1b [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:06:50.270891 2634 raft_consensus.cc:2781] T 59a42d91008d4dffaedbc0a9a2673e99 P a2564232d24e4d959fa3c573a3b9deb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:06:50.271067 2634 raft_consensus.cc:686] T 59a42d91008d4dffaedbc0a9a2673e99 P a2564232d24e4d959fa3c573a3b9deb1 [term 1 LEADER]: Becoming Leader. State: Replica: a2564232d24e4d959fa3c573a3b9deb1, State: Running, Role: LEADER
I20260620 00:06:50.271162 2634 consensus_queue.cc:237] T 59a42d91008d4dffaedbc0a9a2673e99 P a2564232d24e4d959fa3c573a3b9deb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } }
I20260620 00:06:50.271615 2417 raft_consensus.cc:2454] T 59a42d91008d4dffaedbc0a9a2673e99 P 84332eccbe6642c886c8ae3489db3b1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2564232d24e4d959fa3c573a3b9deb1 in term 1.
I20260620 00:06:50.271785 2648 tablet_replica.cc:439] T 59a42d91008d4dffaedbc0a9a2673e99 P a2564232d24e4d959fa3c573a3b9deb1: TxnStatusTablet state changed. Reason: New leader a2564232d24e4d959fa3c573a3b9deb1. Latest consensus state: current_term: 1 leader_uuid: "a2564232d24e4d959fa3c573a3b9deb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } } }
I20260620 00:06:50.271872 2648 tablet_replica.cc:442] T 59a42d91008d4dffaedbc0a9a2673e99 P a2564232d24e4d959fa3c573a3b9deb1: This TxnStatusTablet replica's current role is: LEADER
I20260620 00:06:50.272018 2120 catalog_manager.cc:5495] T 59a42d91008d4dffaedbc0a9a2673e99 P a2564232d24e4d959fa3c573a3b9deb1 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2564232d24e4d959fa3c573a3b9deb1 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a2564232d24e4d959fa3c573a3b9deb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } health_report { overall_health: UNKNOWN } } }
I20260620 00:06:50.272306 2650 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 00:06:50.272440 2650 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 00:06:50.282579 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:50.279817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38762 (local address 127.0.98.1:37291)
0620 00:06:50.279882 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:50.279884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:50.279893 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:06:50.279982 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:50.279983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:50.279997 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:50.280073 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:06:50.280206 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.280666 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.281800 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.281986 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.282042 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:50.282238 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:06:50.282355 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:06:50.282423 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:50.282492 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 00:06:50.282511 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 00:06:50.282624 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:50.279724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37291 (local address 127.0.0.1:38762)
0620 00:06:50.279818 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:50.279833 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:50.279885 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:50.280086 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:50.280087 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:50.280088 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 00:06:50.280140 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:50.280142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.280748 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.280749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:50.281706 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:50.281709 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.282057 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.282058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:50.282127 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:50.282143 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 00:06:50.282423 (+ 280us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 00:06:50.282431 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 00:06:50.282488 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 00:06:50.282541 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 00:06:50.284452 2557 raft_consensus.cc:1261] T 59a42d91008d4dffaedbc0a9a2673e99 P 66516f726b524477910e30b8165b19f2 [term 1 FOLLOWER]: Refusing update from remote peer a2564232d24e4d959fa3c573a3b9deb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:06:50.284474 2417 raft_consensus.cc:1261] T 59a42d91008d4dffaedbc0a9a2673e99 P 84332eccbe6642c886c8ae3489db3b1b [term 1 FOLLOWER]: Refusing update from remote peer a2564232d24e4d959fa3c573a3b9deb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:06:50.284782 2648 consensus_queue.cc:1035] T 59a42d91008d4dffaedbc0a9a2673e99 P a2564232d24e4d959fa3c573a3b9deb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:06:50.284905 2634 consensus_queue.cc:1035] T 59a42d91008d4dffaedbc0a9a2673e99 P a2564232d24e4d959fa3c573a3b9deb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:06:50.286099 2633 tablet_replica.cc:439] T 59a42d91008d4dffaedbc0a9a2673e99 P 84332eccbe6642c886c8ae3489db3b1b: TxnStatusTablet state changed. Reason: New leader a2564232d24e4d959fa3c573a3b9deb1. Latest consensus state: current_term: 1 leader_uuid: "a2564232d24e4d959fa3c573a3b9deb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } } }
I20260620 00:06:50.286192 2633 tablet_replica.cc:442] T 59a42d91008d4dffaedbc0a9a2673e99 P 84332eccbe6642c886c8ae3489db3b1b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:06:50.286150 2632 tablet_replica.cc:439] T 59a42d91008d4dffaedbc0a9a2673e99 P 66516f726b524477910e30b8165b19f2: TxnStatusTablet state changed. Reason: New leader a2564232d24e4d959fa3c573a3b9deb1. Latest consensus state: current_term: 1 leader_uuid: "a2564232d24e4d959fa3c573a3b9deb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } } }
I20260620 00:06:50.286250 2632 tablet_replica.cc:442] T 59a42d91008d4dffaedbc0a9a2673e99 P 66516f726b524477910e30b8165b19f2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:06:50.287050 2634 tablet_replica.cc:439] T 59a42d91008d4dffaedbc0a9a2673e99 P a2564232d24e4d959fa3c573a3b9deb1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a2564232d24e4d959fa3c573a3b9deb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } } }
I20260620 00:06:50.287132 2634 tablet_replica.cc:442] T 59a42d91008d4dffaedbc0a9a2673e99 P a2564232d24e4d959fa3c573a3b9deb1: This TxnStatusTablet replica's current role is: LEADER
I20260620 00:06:50.287190 2653 mvcc.cc:205] Tried to move back new op lower bound from 7298719786122371072 to 7298719786071236608. Current Snapshot: MvccSnapshot[applied={T|T < 7298719786122371072}]
I20260620 00:06:50.287386 2633 tablet_replica.cc:439] T 59a42d91008d4dffaedbc0a9a2673e99 P 84332eccbe6642c886c8ae3489db3b1b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a2564232d24e4d959fa3c573a3b9deb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } } }
I20260620 00:06:50.287386 2634 tablet_replica.cc:439] T 59a42d91008d4dffaedbc0a9a2673e99 P a2564232d24e4d959fa3c573a3b9deb1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a2564232d24e4d959fa3c573a3b9deb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } } }
I20260620 00:06:50.287477 2634 tablet_replica.cc:442] T 59a42d91008d4dffaedbc0a9a2673e99 P a2564232d24e4d959fa3c573a3b9deb1: This TxnStatusTablet replica's current role is: LEADER
I20260620 00:06:50.287477 2633 tablet_replica.cc:442] T 59a42d91008d4dffaedbc0a9a2673e99 P 84332eccbe6642c886c8ae3489db3b1b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:06:50.287752 2632 tablet_replica.cc:439] T 59a42d91008d4dffaedbc0a9a2673e99 P 66516f726b524477910e30b8165b19f2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a2564232d24e4d959fa3c573a3b9deb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } } }
I20260620 00:06:50.287838 2632 tablet_replica.cc:442] T 59a42d91008d4dffaedbc0a9a2673e99 P 66516f726b524477910e30b8165b19f2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:06:50.288175 2655 mvcc.cc:205] Tried to move back new op lower bound from 7298719786122371072 to 7298719786071236608. Current Snapshot: MvccSnapshot[applied={T|T < 7298719786122371072}]
I20260620 00:06:50.288381 2654 mvcc.cc:205] Tried to move back new op lower bound from 7298719786122371072 to 7298719786071236608. Current Snapshot: MvccSnapshot[applied={T|T < 7298719786122371072}]
I20260620 00:06:50.292901 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:50.290283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39788 (local address 127.0.98.2:36571)
0620 00:06:50.290317 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:50.290319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:50.290328 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:06:50.290387 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:50.290388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:50.290401 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:50.290453 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:06:50.290588 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.291072 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.292181 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.292376 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.292436 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:50.292610 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:06:50.292702 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:06:50.292793 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:50.292815 (+ 22us) server_negotiation.cc:299] Negotiation successful
0620 00:06:50.292840 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 00:06:50.297917 2469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:50.295269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37291 (local address 127.0.98.2:37315)
0620 00:06:50.295341 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:50.295354 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:50.295399 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:50.295542 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:50.295543 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:50.295544 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 00:06:50.295596 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:50.295597 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.296446 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.296446 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:50.296987 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:50.296989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.297473 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.297474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:50.297523 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:50.297535 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 00:06:50.297769 (+ 234us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 00:06:50.297771 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 00:06:50.297825 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 00:06:50.297864 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 00:06:50.297959 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:50.295346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37315 (local address 127.0.98.1:37291)
0620 00:06:50.295392 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:50.295394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:50.295404 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:06:50.295464 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:50.295465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:50.295476 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:50.295541 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:06:50.295637 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.296367 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.297066 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.297392 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.297463 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:50.297621 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:06:50.297654 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:06:50.297711 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:50.297843 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 00:06:50.297866 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 00:06:50.300395 2277 raft_consensus.cc:1261] T 5ca4f99f419f4607915f719140fa623e P a2564232d24e4d959fa3c573a3b9deb1 [term 1 FOLLOWER]: Refusing update from remote peer 84332eccbe6642c886c8ae3489db3b1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:06:50.300634 2557 raft_consensus.cc:1261] T 5ca4f99f419f4607915f719140fa623e P 66516f726b524477910e30b8165b19f2 [term 1 FOLLOWER]: Refusing update from remote peer 84332eccbe6642c886c8ae3489db3b1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:06:50.300704 2636 consensus_queue.cc:1035] T 5ca4f99f419f4607915f719140fa623e P 84332eccbe6642c886c8ae3489db3b1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:06:50.300822 2633 consensus_queue.cc:1035] T 5ca4f99f419f4607915f719140fa623e P 84332eccbe6642c886c8ae3489db3b1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:06:50.309266 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:50.306380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41889 (local address 127.0.98.2:36571)
0620 00:06:50.306418 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:50.306420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:50.306578 (+ 158us) server_negotiation.cc:407] Connection header received
0620 00:06:50.306619 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:50.306620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:50.306631 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:50.306691 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:06:50.306781 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.307530 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.308488 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.308779 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.308854 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:50.309038 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:06:50.309071 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:06:50.309132 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:50.309172 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 00:06:50.309191 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 00:06:50.309340 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:50.306332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36571 (local address 127.0.98.1:41889)
0620 00:06:50.306498 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:50.306509 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:50.306569 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:50.306691 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:50.306692 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:50.306693 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 00:06:50.306744 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:50.306745 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.307611 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.307613 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:50.308410 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:50.308412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:50.308856 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:50.308857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:50.308904 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:50.308916 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 00:06:50.309136 (+ 220us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 00:06:50.309137 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 00:06:50.309168 (+ 31us) client_negotiation.cc:240] Negotiation successful
0620 00:06:50.309230 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":102,"threads_started":1}
I20260620 00:06:50.317958 2119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:06:50.318090 2119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:06:50.318825 2119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 49f4add4939e45c4998254385ad73d44: Sending DeleteTablet for 3 replicas of tablet 5ca4f99f419f4607915f719140fa623e
I20260620 00:06:50.319123 2397 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ca4f99f419f4607915f719140fa623e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:06:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39774
I20260620 00:06:50.319263 2537 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ca4f99f419f4607915f719140fa623e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:06:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49828
I20260620 00:06:50.319334 2119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55528:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:06:50.319475 2119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:06:50.319474 2257 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ca4f99f419f4607915f719140fa623e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:06:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38754
I20260620 00:06:50.319525 2674 tablet_replica.cc:330] T 5ca4f99f419f4607915f719140fa623e P 66516f726b524477910e30b8165b19f2: stopping tablet replica
I20260620 00:06:50.319633 2674 raft_consensus.cc:2229] T 5ca4f99f419f4607915f719140fa623e P 66516f726b524477910e30b8165b19f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:06:50.319697 2674 raft_consensus.cc:2258] T 5ca4f99f419f4607915f719140fa623e P 66516f726b524477910e30b8165b19f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:06:50.320272 2674 ts_tablet_manager.cc:1905] T 5ca4f99f419f4607915f719140fa623e P 66516f726b524477910e30b8165b19f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:06:50.321130 2397 tablet_service.cc:1467] Processing CreateTablet for tablet ee4d562569884f5b93d1b578be3709f1 (DEFAULT_TABLE table=test-table [id=9c6cb4847d26421094f066579e0e9d78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:06:50.321254 2397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee4d562569884f5b93d1b578be3709f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:06:50.321493 2537 tablet_service.cc:1467] Processing CreateTablet for tablet ee4d562569884f5b93d1b578be3709f1 (DEFAULT_TABLE table=test-table [id=9c6cb4847d26421094f066579e0e9d78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:06:50.321591 2537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee4d562569884f5b93d1b578be3709f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:06:50.322077 2674 ts_tablet_manager.cc:1918] T 5ca4f99f419f4607915f719140fa623e P 66516f726b524477910e30b8165b19f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 00:06:50.322147 2674 log.cc:1196] T 5ca4f99f419f4607915f719140fa623e P 66516f726b524477910e30b8165b19f2: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-2/wal/wals/5ca4f99f419f4607915f719140fa623e
I20260620 00:06:50.322412 2674 ts_tablet_manager.cc:1939] T 5ca4f99f419f4607915f719140fa623e P 66516f726b524477910e30b8165b19f2: Deleting consensus metadata
I20260620 00:06:50.322778 2626 tablet_bootstrap.cc:492] T ee4d562569884f5b93d1b578be3709f1 P 66516f726b524477910e30b8165b19f2: Bootstrap starting.
I20260620 00:06:50.322831 2105 catalog_manager.cc:4841] TS 66516f726b524477910e30b8165b19f2 (127.0.98.3:38535): tablet 5ca4f99f419f4607915f719140fa623e (table test-table [id=12fec3a50134400b959f307a7f1b799f]) successfully deleted
I20260620 00:06:50.323173 2626 tablet_bootstrap.cc:654] T ee4d562569884f5b93d1b578be3709f1 P 66516f726b524477910e30b8165b19f2: Neither blocks nor log segments found. Creating new log.
I20260620 00:06:50.323515 2257 tablet_service.cc:1467] Processing CreateTablet for tablet ee4d562569884f5b93d1b578be3709f1 (DEFAULT_TABLE table=test-table [id=9c6cb4847d26421094f066579e0e9d78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:06:50.323623 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee4d562569884f5b93d1b578be3709f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:06:50.324028 2627 tablet_bootstrap.cc:492] T ee4d562569884f5b93d1b578be3709f1 P 84332eccbe6642c886c8ae3489db3b1b: Bootstrap starting.
I20260620 00:06:50.324460 2627 tablet_bootstrap.cc:654] T ee4d562569884f5b93d1b578be3709f1 P 84332eccbe6642c886c8ae3489db3b1b: Neither blocks nor log segments found. Creating new log.
I20260620 00:06:50.324903 2628 tablet_bootstrap.cc:492] T ee4d562569884f5b93d1b578be3709f1 P a2564232d24e4d959fa3c573a3b9deb1: Bootstrap starting.
I20260620 00:06:50.325163 2626 tablet_bootstrap.cc:492] T ee4d562569884f5b93d1b578be3709f1 P 66516f726b524477910e30b8165b19f2: No bootstrap required, opened a new log
I20260620 00:06:50.325214 2626 ts_tablet_manager.cc:1397] T ee4d562569884f5b93d1b578be3709f1 P 66516f726b524477910e30b8165b19f2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:06:50.325224 2627 tablet_bootstrap.cc:492] T ee4d562569884f5b93d1b578be3709f1 P 84332eccbe6642c886c8ae3489db3b1b: No bootstrap required, opened a new log
I20260620 00:06:50.325273 2627 ts_tablet_manager.cc:1397] T ee4d562569884f5b93d1b578be3709f1 P 84332eccbe6642c886c8ae3489db3b1b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:06:50.325356 2626 raft_consensus.cc:348] T ee4d562569884f5b93d1b578be3709f1 P 66516f726b524477910e30b8165b19f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } }
I20260620 00:06:50.325446 2626 raft_consensus.cc:374] T ee4d562569884f5b93d1b578be3709f1 P 66516f726b524477910e30b8165b19f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:06:50.325467 2626 raft_consensus.cc:729] T ee4d562569884f5b93d1b578be3709f1 P 66516f726b524477910e30b8165b19f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66516f726b524477910e30b8165b19f2, State: Initialized, Role: FOLLOWER
I20260620 00:06:50.325505 2628 tablet_bootstrap.cc:654] T ee4d562569884f5b93d1b578be3709f1 P a2564232d24e4d959fa3c573a3b9deb1: Neither blocks nor log segments found. Creating new log.
I20260620 00:06:50.325469 2627 raft_consensus.cc:348] T ee4d562569884f5b93d1b578be3709f1 P 84332eccbe6642c886c8ae3489db3b1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } }
I20260620 00:06:50.325522 2626 consensus_queue.cc:260] T ee4d562569884f5b93d1b578be3709f1 P 66516f726b524477910e30b8165b19f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } }
I20260620 00:06:50.325543 2627 raft_consensus.cc:374] T ee4d562569884f5b93d1b578be3709f1 P 84332eccbe6642c886c8ae3489db3b1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:06:50.325583 2627 raft_consensus.cc:729] T ee4d562569884f5b93d1b578be3709f1 P 84332eccbe6642c886c8ae3489db3b1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84332eccbe6642c886c8ae3489db3b1b, State: Initialized, Role: FOLLOWER
I20260620 00:06:50.325609 2626 ts_tablet_manager.cc:1428] T ee4d562569884f5b93d1b578be3709f1 P 66516f726b524477910e30b8165b19f2: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 00:06:50.325635 2627 consensus_queue.cc:260] T ee4d562569884f5b93d1b578be3709f1 P 84332eccbe6642c886c8ae3489db3b1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } }
I20260620 00:06:50.325757 2627 ts_tablet_manager.cc:1428] T ee4d562569884f5b93d1b578be3709f1 P 84332eccbe6642c886c8ae3489db3b1b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:50.326104 2628 tablet_bootstrap.cc:492] T ee4d562569884f5b93d1b578be3709f1 P a2564232d24e4d959fa3c573a3b9deb1: No bootstrap required, opened a new log
I20260620 00:06:50.326171 2628 ts_tablet_manager.cc:1397] T ee4d562569884f5b93d1b578be3709f1 P a2564232d24e4d959fa3c573a3b9deb1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:06:50.326311 2628 raft_consensus.cc:348] T ee4d562569884f5b93d1b578be3709f1 P a2564232d24e4d959fa3c573a3b9deb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } }
I20260620 00:06:50.326377 2628 raft_consensus.cc:374] T ee4d562569884f5b93d1b578be3709f1 P a2564232d24e4d959fa3c573a3b9deb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:06:50.326401 2628 raft_consensus.cc:729] T ee4d562569884f5b93d1b578be3709f1 P a2564232d24e4d959fa3c573a3b9deb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2564232d24e4d959fa3c573a3b9deb1, State: Initialized, Role: FOLLOWER
I20260620 00:06:50.326437 2628 consensus_queue.cc:260] T ee4d562569884f5b93d1b578be3709f1 P a2564232d24e4d959fa3c573a3b9deb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } }
I20260620 00:06:50.326531 2628 ts_tablet_manager.cc:1428] T ee4d562569884f5b93d1b578be3709f1 P a2564232d24e4d959fa3c573a3b9deb1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:50.329486 2675 tablet_replica.cc:330] T 5ca4f99f419f4607915f719140fa623e P a2564232d24e4d959fa3c573a3b9deb1: stopping tablet replica
I20260620 00:06:50.329619 2675 raft_consensus.cc:2229] T 5ca4f99f419f4607915f719140fa623e P a2564232d24e4d959fa3c573a3b9deb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:06:50.329681 2675 raft_consensus.cc:2258] T 5ca4f99f419f4607915f719140fa623e P a2564232d24e4d959fa3c573a3b9deb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:06:50.329855 2673 tablet_replica.cc:330] T 5ca4f99f419f4607915f719140fa623e P 84332eccbe6642c886c8ae3489db3b1b: stopping tablet replica
I20260620 00:06:50.329964 2673 raft_consensus.cc:2229] T 5ca4f99f419f4607915f719140fa623e P 84332eccbe6642c886c8ae3489db3b1b [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:06:50.330144 2673 raft_consensus.cc:2258] T 5ca4f99f419f4607915f719140fa623e P 84332eccbe6642c886c8ae3489db3b1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:06:50.330361 2675 ts_tablet_manager.cc:1905] T 5ca4f99f419f4607915f719140fa623e P a2564232d24e4d959fa3c573a3b9deb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:06:50.330546 2673 ts_tablet_manager.cc:1905] T 5ca4f99f419f4607915f719140fa623e P 84332eccbe6642c886c8ae3489db3b1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:06:50.332329 2675 ts_tablet_manager.cc:1918] T 5ca4f99f419f4607915f719140fa623e P a2564232d24e4d959fa3c573a3b9deb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 00:06:50.332276 2673 ts_tablet_manager.cc:1918] T 5ca4f99f419f4607915f719140fa623e P 84332eccbe6642c886c8ae3489db3b1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 00:06:50.332396 2673 log.cc:1196] T 5ca4f99f419f4607915f719140fa623e P 84332eccbe6642c886c8ae3489db3b1b: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-1/wal/wals/5ca4f99f419f4607915f719140fa623e
I20260620 00:06:50.332477 2675 log.cc:1196] T 5ca4f99f419f4607915f719140fa623e P a2564232d24e4d959fa3c573a3b9deb1: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-0/wal/wals/5ca4f99f419f4607915f719140fa623e
I20260620 00:06:50.332683 2673 ts_tablet_manager.cc:1939] T 5ca4f99f419f4607915f719140fa623e P 84332eccbe6642c886c8ae3489db3b1b: Deleting consensus metadata
I20260620 00:06:50.332834 2675 ts_tablet_manager.cc:1939] T 5ca4f99f419f4607915f719140fa623e P a2564232d24e4d959fa3c573a3b9deb1: Deleting consensus metadata
I20260620 00:06:50.333096 2105 catalog_manager.cc:4841] TS 84332eccbe6642c886c8ae3489db3b1b (127.0.98.2:36571): tablet 5ca4f99f419f4607915f719140fa623e (table test-table [id=12fec3a50134400b959f307a7f1b799f]) successfully deleted
I20260620 00:06:50.333413 2105 catalog_manager.cc:4841] TS a2564232d24e4d959fa3c573a3b9deb1 (127.0.98.1:37291): tablet 5ca4f99f419f4607915f719140fa623e (table test-table [id=12fec3a50134400b959f307a7f1b799f]) successfully deleted
W20260620 00:06:50.339641 2604 tablet.cc:2369] T ee4d562569884f5b93d1b578be3709f1 P 66516f726b524477910e30b8165b19f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:06:50.569892 2670 raft_consensus.cc:482] T ee4d562569884f5b93d1b578be3709f1 P 84332eccbe6642c886c8ae3489db3b1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:06:50.569999 2670 raft_consensus.cc:504] T ee4d562569884f5b93d1b578be3709f1 P 84332eccbe6642c886c8ae3489db3b1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } }
I20260620 00:06:50.570148 2670 leader_election.cc:288] T ee4d562569884f5b93d1b578be3709f1 P 84332eccbe6642c886c8ae3489db3b1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2564232d24e4d959fa3c573a3b9deb1 (127.0.98.1:37291), 66516f726b524477910e30b8165b19f2 (127.0.98.3:38535)
I20260620 00:06:50.570458 2557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee4d562569884f5b93d1b578be3709f1" candidate_uuid: "84332eccbe6642c886c8ae3489db3b1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66516f726b524477910e30b8165b19f2" is_pre_election: true
I20260620 00:06:50.570432 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee4d562569884f5b93d1b578be3709f1" candidate_uuid: "84332eccbe6642c886c8ae3489db3b1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2564232d24e4d959fa3c573a3b9deb1" is_pre_election: true
I20260620 00:06:50.570562 2277 raft_consensus.cc:2454] T ee4d562569884f5b93d1b578be3709f1 P a2564232d24e4d959fa3c573a3b9deb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84332eccbe6642c886c8ae3489db3b1b in term 0.
I20260620 00:06:50.570564 2557 raft_consensus.cc:2454] T ee4d562569884f5b93d1b578be3709f1 P 66516f726b524477910e30b8165b19f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84332eccbe6642c886c8ae3489db3b1b in term 0.
I20260620 00:06:50.570755 2349 leader_election.cc:302] T ee4d562569884f5b93d1b578be3709f1 P 84332eccbe6642c886c8ae3489db3b1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84332eccbe6642c886c8ae3489db3b1b, a2564232d24e4d959fa3c573a3b9deb1; no voters:
I20260620 00:06:50.570873 2670 raft_consensus.cc:2781] T ee4d562569884f5b93d1b578be3709f1 P 84332eccbe6642c886c8ae3489db3b1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:06:50.570940 2670 raft_consensus.cc:482] T ee4d562569884f5b93d1b578be3709f1 P 84332eccbe6642c886c8ae3489db3b1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:06:50.570972 2670 raft_consensus.cc:3037] T ee4d562569884f5b93d1b578be3709f1 P 84332eccbe6642c886c8ae3489db3b1b [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:06:50.571727 2670 raft_consensus.cc:504] T ee4d562569884f5b93d1b578be3709f1 P 84332eccbe6642c886c8ae3489db3b1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } }
I20260620 00:06:50.571874 2670 leader_election.cc:288] T ee4d562569884f5b93d1b578be3709f1 P 84332eccbe6642c886c8ae3489db3b1b [CANDIDATE]: Term 1 election: Requested vote from peers a2564232d24e4d959fa3c573a3b9deb1 (127.0.98.1:37291), 66516f726b524477910e30b8165b19f2 (127.0.98.3:38535)
I20260620 00:06:50.572095 2557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee4d562569884f5b93d1b578be3709f1" candidate_uuid: "84332eccbe6642c886c8ae3489db3b1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66516f726b524477910e30b8165b19f2"
I20260620 00:06:50.572093 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee4d562569884f5b93d1b578be3709f1" candidate_uuid: "84332eccbe6642c886c8ae3489db3b1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2564232d24e4d959fa3c573a3b9deb1"
I20260620 00:06:50.572183 2557 raft_consensus.cc:3037] T ee4d562569884f5b93d1b578be3709f1 P 66516f726b524477910e30b8165b19f2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:06:50.572184 2277 raft_consensus.cc:3037] T ee4d562569884f5b93d1b578be3709f1 P a2564232d24e4d959fa3c573a3b9deb1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:06:50.572971 2277 raft_consensus.cc:2454] T ee4d562569884f5b93d1b578be3709f1 P a2564232d24e4d959fa3c573a3b9deb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84332eccbe6642c886c8ae3489db3b1b in term 1.
I20260620 00:06:50.572971 2557 raft_consensus.cc:2454] T ee4d562569884f5b93d1b578be3709f1 P 66516f726b524477910e30b8165b19f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84332eccbe6642c886c8ae3489db3b1b in term 1.
I20260620 00:06:50.573182 2349 leader_election.cc:302] T ee4d562569884f5b93d1b578be3709f1 P 84332eccbe6642c886c8ae3489db3b1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84332eccbe6642c886c8ae3489db3b1b, a2564232d24e4d959fa3c573a3b9deb1; no voters:
I20260620 00:06:50.573293 2670 raft_consensus.cc:2781] T ee4d562569884f5b93d1b578be3709f1 P 84332eccbe6642c886c8ae3489db3b1b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:06:50.573386 2670 raft_consensus.cc:686] T ee4d562569884f5b93d1b578be3709f1 P 84332eccbe6642c886c8ae3489db3b1b [term 1 LEADER]: Becoming Leader. State: Replica: 84332eccbe6642c886c8ae3489db3b1b, State: Running, Role: LEADER
I20260620 00:06:50.573452 2670 consensus_queue.cc:237] T ee4d562569884f5b93d1b578be3709f1 P 84332eccbe6642c886c8ae3489db3b1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } }
I20260620 00:06:50.574092 2119 catalog_manager.cc:5495] T ee4d562569884f5b93d1b578be3709f1 P 84332eccbe6642c886c8ae3489db3b1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 84332eccbe6642c886c8ae3489db3b1b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "84332eccbe6642c886c8ae3489db3b1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } health_report { overall_health: UNKNOWN } } }
I20260620 00:06:50.612922 2277 raft_consensus.cc:1261] T ee4d562569884f5b93d1b578be3709f1 P a2564232d24e4d959fa3c573a3b9deb1 [term 1 FOLLOWER]: Refusing update from remote peer 84332eccbe6642c886c8ae3489db3b1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:06:50.612908 2557 raft_consensus.cc:1261] T ee4d562569884f5b93d1b578be3709f1 P 66516f726b524477910e30b8165b19f2 [term 1 FOLLOWER]: Refusing update from remote peer 84332eccbe6642c886c8ae3489db3b1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:06:50.613227 2670 consensus_queue.cc:1035] T ee4d562569884f5b93d1b578be3709f1 P 84332eccbe6642c886c8ae3489db3b1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:06:50.613335 2667 consensus_queue.cc:1035] T ee4d562569884f5b93d1b578be3709f1 P 84332eccbe6642c886c8ae3489db3b1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 00:06:50.617467 2654 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 00:06:52.135763 2120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:06:52.135874 2120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:06:52.136936 2120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 49f4add4939e45c4998254385ad73d44: Sending DeleteTablet for 3 replicas of tablet ee4d562569884f5b93d1b578be3709f1
I20260620 00:06:52.137317 2397 tablet_service.cc:1514] Processing DeleteTablet for tablet ee4d562569884f5b93d1b578be3709f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:06:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39774
I20260620 00:06:52.137553 2256 tablet_service.cc:1514] Processing DeleteTablet for tablet ee4d562569884f5b93d1b578be3709f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:06:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38754
I20260620 00:06:52.137647 2703 tablet_replica.cc:330] T ee4d562569884f5b93d1b578be3709f1 P 84332eccbe6642c886c8ae3489db3b1b: stopping tablet replica
I20260620 00:06:52.137718 2703 raft_consensus.cc:2229] T ee4d562569884f5b93d1b578be3709f1 P 84332eccbe6642c886c8ae3489db3b1b [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:06:52.137801 2703 raft_consensus.cc:2258] T ee4d562569884f5b93d1b578be3709f1 P 84332eccbe6642c886c8ae3489db3b1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:06:52.137801 2704 tablet_replica.cc:330] T ee4d562569884f5b93d1b578be3709f1 P a2564232d24e4d959fa3c573a3b9deb1: stopping tablet replica
I20260620 00:06:52.137856 2704 raft_consensus.cc:2229] T ee4d562569884f5b93d1b578be3709f1 P a2564232d24e4d959fa3c573a3b9deb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:06:52.137933 2704 raft_consensus.cc:2258] T ee4d562569884f5b93d1b578be3709f1 P a2564232d24e4d959fa3c573a3b9deb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:06:52.138239 2704 ts_tablet_manager.cc:1905] T ee4d562569884f5b93d1b578be3709f1 P a2564232d24e4d959fa3c573a3b9deb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:06:52.138250 2703 ts_tablet_manager.cc:1905] T ee4d562569884f5b93d1b578be3709f1 P 84332eccbe6642c886c8ae3489db3b1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:06:52.139426 2537 tablet_service.cc:1514] Processing DeleteTablet for tablet ee4d562569884f5b93d1b578be3709f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:06:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49828
I20260620 00:06:52.139705 2706 tablet_replica.cc:330] T ee4d562569884f5b93d1b578be3709f1 P 66516f726b524477910e30b8165b19f2: stopping tablet replica
I20260620 00:06:52.139791 2706 raft_consensus.cc:2229] T ee4d562569884f5b93d1b578be3709f1 P 66516f726b524477910e30b8165b19f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:06:52.139842 2706 raft_consensus.cc:2258] T ee4d562569884f5b93d1b578be3709f1 P 66516f726b524477910e30b8165b19f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:06:52.140769 2706 ts_tablet_manager.cc:1905] T ee4d562569884f5b93d1b578be3709f1 P 66516f726b524477910e30b8165b19f2: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 00:06:52 dist-test-slave-grqt krb5kdc[2075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914012, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260620 00:06:52.166944 2706 ts_tablet_manager.cc:1918] T ee4d562569884f5b93d1b578be3709f1 P 66516f726b524477910e30b8165b19f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 00:06:52.167042 2706 log.cc:1196] T ee4d562569884f5b93d1b578be3709f1 P 66516f726b524477910e30b8165b19f2: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-2/wal/wals/ee4d562569884f5b93d1b578be3709f1
I20260620 00:06:52.167315 2706 ts_tablet_manager.cc:1939] T ee4d562569884f5b93d1b578be3709f1 P 66516f726b524477910e30b8165b19f2: Deleting consensus metadata
I20260620 00:06:52.167644 2703 ts_tablet_manager.cc:1918] T ee4d562569884f5b93d1b578be3709f1 P 84332eccbe6642c886c8ae3489db3b1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 00:06:52.167721 2703 log.cc:1196] T ee4d562569884f5b93d1b578be3709f1 P 84332eccbe6642c886c8ae3489db3b1b: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-1/wal/wals/ee4d562569884f5b93d1b578be3709f1
I20260620 00:06:52.168028 2703 ts_tablet_manager.cc:1939] T ee4d562569884f5b93d1b578be3709f1 P 84332eccbe6642c886c8ae3489db3b1b: Deleting consensus metadata
Jun 20 00:06:52 dist-test-slave-grqt krb5kdc[2075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914012, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:06:52.175454 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:52.165886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55552 (local address 127.0.98.62:40575)
0620 00:06:52.166075 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:52.166077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:52.166160 (+ 83us) server_negotiation.cc:407] Connection header received
0620 00:06:52.166248 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:52.166249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:52.166280 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:52.166365 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:52.167288 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:52.168201 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:52.169511 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:52.169798 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:52.169870 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:52.172667 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:52.172668 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:52.172669 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:52.172676 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:52.174212 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:52.174613 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:52.174615 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:52.174615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:52.174700 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:52.174900 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:52.174901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:52.174901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:52.175053 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:52.175133 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:52.175271 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 00:06:52.175343 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":110,"threads_started":1}
I20260620 00:06:52.177281 2120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55552:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:06:52.177527 2120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:06:52.179103 2537 tablet_service.cc:1467] Processing CreateTablet for tablet 0a68b775e3784f2c8544dba86505c86f (DEFAULT_TABLE table=test-table [id=aec9e39fc81c418ebdbf6884f9c2c2fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:06:52.179111 2256 tablet_service.cc:1467] Processing CreateTablet for tablet 0a68b775e3784f2c8544dba86505c86f (DEFAULT_TABLE table=test-table [id=aec9e39fc81c418ebdbf6884f9c2c2fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:06:52.179272 2537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a68b775e3784f2c8544dba86505c86f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:06:52.179287 2397 tablet_service.cc:1467] Processing CreateTablet for tablet 0a68b775e3784f2c8544dba86505c86f (DEFAULT_TABLE table=test-table [id=aec9e39fc81c418ebdbf6884f9c2c2fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:06:52.179423 2397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a68b775e3784f2c8544dba86505c86f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:06:52.180577 2714 tablet_bootstrap.cc:492] T 0a68b775e3784f2c8544dba86505c86f P 84332eccbe6642c886c8ae3489db3b1b: Bootstrap starting.
I20260620 00:06:52.180685 2715 tablet_bootstrap.cc:492] T 0a68b775e3784f2c8544dba86505c86f P 66516f726b524477910e30b8165b19f2: Bootstrap starting.
I20260620 00:06:52.181032 2714 tablet_bootstrap.cc:654] T 0a68b775e3784f2c8544dba86505c86f P 84332eccbe6642c886c8ae3489db3b1b: Neither blocks nor log segments found. Creating new log.
I20260620 00:06:52.181123 2715 tablet_bootstrap.cc:654] T 0a68b775e3784f2c8544dba86505c86f P 66516f726b524477910e30b8165b19f2: Neither blocks nor log segments found. Creating new log.
I20260620 00:06:52.181682 2714 tablet_bootstrap.cc:492] T 0a68b775e3784f2c8544dba86505c86f P 84332eccbe6642c886c8ae3489db3b1b: No bootstrap required, opened a new log
I20260620 00:06:52.181739 2714 ts_tablet_manager.cc:1397] T 0a68b775e3784f2c8544dba86505c86f P 84332eccbe6642c886c8ae3489db3b1b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:06:52.181869 2715 tablet_bootstrap.cc:492] T 0a68b775e3784f2c8544dba86505c86f P 66516f726b524477910e30b8165b19f2: No bootstrap required, opened a new log
I20260620 00:06:52.181951 2715 ts_tablet_manager.cc:1397] T 0a68b775e3784f2c8544dba86505c86f P 66516f726b524477910e30b8165b19f2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:06:52.181893 2714 raft_consensus.cc:348] T 0a68b775e3784f2c8544dba86505c86f P 84332eccbe6642c886c8ae3489db3b1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } }
I20260620 00:06:52.182042 2714 raft_consensus.cc:374] T 0a68b775e3784f2c8544dba86505c86f P 84332eccbe6642c886c8ae3489db3b1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:06:52.182077 2714 raft_consensus.cc:729] T 0a68b775e3784f2c8544dba86505c86f P 84332eccbe6642c886c8ae3489db3b1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84332eccbe6642c886c8ae3489db3b1b, State: Initialized, Role: FOLLOWER
I20260620 00:06:52.182113 2715 raft_consensus.cc:348] T 0a68b775e3784f2c8544dba86505c86f P 66516f726b524477910e30b8165b19f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } }
I20260620 00:06:52.182144 2714 consensus_queue.cc:260] T 0a68b775e3784f2c8544dba86505c86f P 84332eccbe6642c886c8ae3489db3b1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } }
I20260620 00:06:52.182185 2715 raft_consensus.cc:374] T 0a68b775e3784f2c8544dba86505c86f P 66516f726b524477910e30b8165b19f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:06:52.182210 2715 raft_consensus.cc:729] T 0a68b775e3784f2c8544dba86505c86f P 66516f726b524477910e30b8165b19f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66516f726b524477910e30b8165b19f2, State: Initialized, Role: FOLLOWER
I20260620 00:06:52.182251 2714 ts_tablet_manager.cc:1428] T 0a68b775e3784f2c8544dba86505c86f P 84332eccbe6642c886c8ae3489db3b1b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:52.182284 2715 consensus_queue.cc:260] T 0a68b775e3784f2c8544dba86505c86f P 66516f726b524477910e30b8165b19f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } }
I20260620 00:06:52.182456 2715 ts_tablet_manager.cc:1428] T 0a68b775e3784f2c8544dba86505c86f P 66516f726b524477910e30b8165b19f2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 00:06:52.184425 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a68b775e3784f2c8544dba86505c86f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:06:52.184437 2704 ts_tablet_manager.cc:1918] T ee4d562569884f5b93d1b578be3709f1 P a2564232d24e4d959fa3c573a3b9deb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 00:06:52.184554 2704 log.cc:1196] T ee4d562569884f5b93d1b578be3709f1 P a2564232d24e4d959fa3c573a3b9deb1: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-0/wal/wals/ee4d562569884f5b93d1b578be3709f1
I20260620 00:06:52.184794 2704 ts_tablet_manager.cc:1939] T ee4d562569884f5b93d1b578be3709f1 P a2564232d24e4d959fa3c573a3b9deb1: Deleting consensus metadata
I20260620 00:06:52.186004 2719 tablet_bootstrap.cc:492] T 0a68b775e3784f2c8544dba86505c86f P a2564232d24e4d959fa3c573a3b9deb1: Bootstrap starting.
I20260620 00:06:52.186393 2719 tablet_bootstrap.cc:654] T 0a68b775e3784f2c8544dba86505c86f P a2564232d24e4d959fa3c573a3b9deb1: Neither blocks nor log segments found. Creating new log.
I20260620 00:06:52.186976 2719 tablet_bootstrap.cc:492] T 0a68b775e3784f2c8544dba86505c86f P a2564232d24e4d959fa3c573a3b9deb1: No bootstrap required, opened a new log
I20260620 00:06:52.187067 2719 ts_tablet_manager.cc:1397] T 0a68b775e3784f2c8544dba86505c86f P a2564232d24e4d959fa3c573a3b9deb1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:06:52.187181 2719 raft_consensus.cc:348] T 0a68b775e3784f2c8544dba86505c86f P a2564232d24e4d959fa3c573a3b9deb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } }
I20260620 00:06:52.187268 2719 raft_consensus.cc:374] T 0a68b775e3784f2c8544dba86505c86f P a2564232d24e4d959fa3c573a3b9deb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:06:52.187299 2719 raft_consensus.cc:729] T 0a68b775e3784f2c8544dba86505c86f P a2564232d24e4d959fa3c573a3b9deb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2564232d24e4d959fa3c573a3b9deb1, State: Initialized, Role: FOLLOWER
I20260620 00:06:52.187373 2719 consensus_queue.cc:260] T 0a68b775e3784f2c8544dba86505c86f P a2564232d24e4d959fa3c573a3b9deb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } }
I20260620 00:06:52.187497 2719 ts_tablet_manager.cc:1428] T 0a68b775e3784f2c8544dba86505c86f P a2564232d24e4d959fa3c573a3b9deb1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 00:06:52.188833 2105 catalog_manager.cc:4841] TS 66516f726b524477910e30b8165b19f2 (127.0.98.3:38535): tablet ee4d562569884f5b93d1b578be3709f1 (table test-table [id=9c6cb4847d26421094f066579e0e9d78]) successfully deleted
I20260620 00:06:52.189726 2105 catalog_manager.cc:4841] TS a2564232d24e4d959fa3c573a3b9deb1 (127.0.98.1:37291): tablet ee4d562569884f5b93d1b578be3709f1 (table test-table [id=9c6cb4847d26421094f066579e0e9d78]) successfully deleted
I20260620 00:06:52.219265 2105 catalog_manager.cc:4841] TS 84332eccbe6642c886c8ae3489db3b1b (127.0.98.2:36571): tablet ee4d562569884f5b93d1b578be3709f1 (table test-table [id=9c6cb4847d26421094f066579e0e9d78]) successfully deleted
I20260620 00:06:52.361230 2664 raft_consensus.cc:482] T 0a68b775e3784f2c8544dba86505c86f P a2564232d24e4d959fa3c573a3b9deb1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:06:52.361490 2664 raft_consensus.cc:504] T 0a68b775e3784f2c8544dba86505c86f P a2564232d24e4d959fa3c573a3b9deb1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } }
I20260620 00:06:52.361703 2664 leader_election.cc:288] T 0a68b775e3784f2c8544dba86505c86f P a2564232d24e4d959fa3c573a3b9deb1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84332eccbe6642c886c8ae3489db3b1b (127.0.98.2:36571), 66516f726b524477910e30b8165b19f2 (127.0.98.3:38535)
I20260620 00:06:52.361948 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a68b775e3784f2c8544dba86505c86f" candidate_uuid: "a2564232d24e4d959fa3c573a3b9deb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84332eccbe6642c886c8ae3489db3b1b" is_pre_election: true
I20260620 00:06:52.361972 2557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a68b775e3784f2c8544dba86505c86f" candidate_uuid: "a2564232d24e4d959fa3c573a3b9deb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66516f726b524477910e30b8165b19f2" is_pre_election: true
I20260620 00:06:52.362063 2417 raft_consensus.cc:2454] T 0a68b775e3784f2c8544dba86505c86f P 84332eccbe6642c886c8ae3489db3b1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2564232d24e4d959fa3c573a3b9deb1 in term 0.
I20260620 00:06:52.362094 2557 raft_consensus.cc:2454] T 0a68b775e3784f2c8544dba86505c86f P 66516f726b524477910e30b8165b19f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2564232d24e4d959fa3c573a3b9deb1 in term 0.
I20260620 00:06:52.362257 2209 leader_election.cc:302] T 0a68b775e3784f2c8544dba86505c86f P a2564232d24e4d959fa3c573a3b9deb1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84332eccbe6642c886c8ae3489db3b1b, a2564232d24e4d959fa3c573a3b9deb1; no voters:
I20260620 00:06:52.362351 2664 raft_consensus.cc:2781] T 0a68b775e3784f2c8544dba86505c86f P a2564232d24e4d959fa3c573a3b9deb1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:06:52.362387 2664 raft_consensus.cc:482] T 0a68b775e3784f2c8544dba86505c86f P a2564232d24e4d959fa3c573a3b9deb1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:06:52.362416 2664 raft_consensus.cc:3037] T 0a68b775e3784f2c8544dba86505c86f P a2564232d24e4d959fa3c573a3b9deb1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:06:52.363364 2664 raft_consensus.cc:504] T 0a68b775e3784f2c8544dba86505c86f P a2564232d24e4d959fa3c573a3b9deb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } }
I20260620 00:06:52.363494 2664 leader_election.cc:288] T 0a68b775e3784f2c8544dba86505c86f P a2564232d24e4d959fa3c573a3b9deb1 [CANDIDATE]: Term 1 election: Requested vote from peers 84332eccbe6642c886c8ae3489db3b1b (127.0.98.2:36571), 66516f726b524477910e30b8165b19f2 (127.0.98.3:38535)
I20260620 00:06:52.363687 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a68b775e3784f2c8544dba86505c86f" candidate_uuid: "a2564232d24e4d959fa3c573a3b9deb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84332eccbe6642c886c8ae3489db3b1b"
I20260620 00:06:52.363690 2557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a68b775e3784f2c8544dba86505c86f" candidate_uuid: "a2564232d24e4d959fa3c573a3b9deb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66516f726b524477910e30b8165b19f2"
I20260620 00:06:52.363771 2417 raft_consensus.cc:3037] T 0a68b775e3784f2c8544dba86505c86f P 84332eccbe6642c886c8ae3489db3b1b [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:06:52.363770 2557 raft_consensus.cc:3037] T 0a68b775e3784f2c8544dba86505c86f P 66516f726b524477910e30b8165b19f2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:06:52.364419 2557 raft_consensus.cc:2454] T 0a68b775e3784f2c8544dba86505c86f P 66516f726b524477910e30b8165b19f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2564232d24e4d959fa3c573a3b9deb1 in term 1.
I20260620 00:06:52.364566 2417 raft_consensus.cc:2454] T 0a68b775e3784f2c8544dba86505c86f P 84332eccbe6642c886c8ae3489db3b1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2564232d24e4d959fa3c573a3b9deb1 in term 1.
I20260620 00:06:52.364602 2209 leader_election.cc:302] T 0a68b775e3784f2c8544dba86505c86f P a2564232d24e4d959fa3c573a3b9deb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66516f726b524477910e30b8165b19f2, a2564232d24e4d959fa3c573a3b9deb1; no voters:
I20260620 00:06:52.364681 2664 raft_consensus.cc:2781] T 0a68b775e3784f2c8544dba86505c86f P a2564232d24e4d959fa3c573a3b9deb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:06:52.364734 2664 raft_consensus.cc:686] T 0a68b775e3784f2c8544dba86505c86f P a2564232d24e4d959fa3c573a3b9deb1 [term 1 LEADER]: Becoming Leader. State: Replica: a2564232d24e4d959fa3c573a3b9deb1, State: Running, Role: LEADER
I20260620 00:06:52.364784 2664 consensus_queue.cc:237] T 0a68b775e3784f2c8544dba86505c86f P a2564232d24e4d959fa3c573a3b9deb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } }
I20260620 00:06:52.365417 2119 catalog_manager.cc:5495] T 0a68b775e3784f2c8544dba86505c86f P a2564232d24e4d959fa3c573a3b9deb1 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2564232d24e4d959fa3c573a3b9deb1 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a2564232d24e4d959fa3c573a3b9deb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } health_report { overall_health: UNKNOWN } } }
I20260620 00:06:52.414773 2723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:52.412294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38776 (local address 127.0.98.1:37291)
0620 00:06:52.412437 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:52.412439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:52.412445 (+ 6us) server_negotiation.cc:407] Connection header received
0620 00:06:52.412463 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:52.412463 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:52.412473 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:52.412530 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:06:52.412769 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:52.413258 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:52.414130 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:52.414324 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:52.414383 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:52.414510 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:06:52.414537 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:06:52.414574 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:52.414643 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 00:06:52.414659 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":91,"threads_started":1}
I20260620 00:06:52.419652 2723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:52.417332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54871 (local address 127.0.98.1:37291)
0620 00:06:52.417391 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:52.417393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:52.417400 (+ 7us) server_negotiation.cc:407] Connection header received
0620 00:06:52.417423 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:52.417424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:52.417431 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:52.417487 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:06:52.417627 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:52.418086 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:52.418959 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:52.419149 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:52.419233 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:52.419374 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:06:52.419410 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:06:52.419470 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:52.419523 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 00:06:52.419557 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":43}
I20260620 00:06:52.419654 2726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:52.417123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37291 (local address 127.0.98.1:54871)
0620 00:06:52.417261 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:52.417276 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:52.417316 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:52.417502 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:52.417503 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:52.417504 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 00:06:52.417553 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:52.417555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:52.418177 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:52.418178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:52.418827 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:52.418829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:52.419230 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:52.419231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:52.419282 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:52.419285 (+ 3us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 00:06:52.419471 (+ 186us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 00:06:52.419473 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 00:06:52.419521 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 00:06:52.419584 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":17,"thread_start_us":87,"threads_started":1}
I20260620 00:06:52.421991 2557 raft_consensus.cc:1261] T 0a68b775e3784f2c8544dba86505c86f P 66516f726b524477910e30b8165b19f2 [term 1 FOLLOWER]: Refusing update from remote peer a2564232d24e4d959fa3c573a3b9deb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:06:52.422039 2417 raft_consensus.cc:1261] T 0a68b775e3784f2c8544dba86505c86f P 84332eccbe6642c886c8ae3489db3b1b [term 1 FOLLOWER]: Refusing update from remote peer a2564232d24e4d959fa3c573a3b9deb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:06:52.422241 2684 consensus_queue.cc:1035] T 0a68b775e3784f2c8544dba86505c86f P a2564232d24e4d959fa3c573a3b9deb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:06:52.422381 2664 consensus_queue.cc:1035] T 0a68b775e3784f2c8544dba86505c86f P a2564232d24e4d959fa3c573a3b9deb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:06:52.433679 2119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:06:52.433811 2119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:06:52.434476 2119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 49f4add4939e45c4998254385ad73d44: Sending DeleteTablet for 3 replicas of tablet 0a68b775e3784f2c8544dba86505c86f
I20260620 00:06:52.434780 2257 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a68b775e3784f2c8544dba86505c86f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:06:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38754
I20260620 00:06:52.434809 2397 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a68b775e3784f2c8544dba86505c86f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:06:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39774
I20260620 00:06:52.434825 2537 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a68b775e3784f2c8544dba86505c86f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:06:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49828
I20260620 00:06:52.434878 2704 tablet_replica.cc:330] T 0a68b775e3784f2c8544dba86505c86f P a2564232d24e4d959fa3c573a3b9deb1: stopping tablet replica
I20260620 00:06:52.434929 2704 raft_consensus.cc:2229] T 0a68b775e3784f2c8544dba86505c86f P a2564232d24e4d959fa3c573a3b9deb1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:06:52.434927 2703 tablet_replica.cc:330] T 0a68b775e3784f2c8544dba86505c86f P 84332eccbe6642c886c8ae3489db3b1b: stopping tablet replica
I20260620 00:06:52.434984 2703 raft_consensus.cc:2229] T 0a68b775e3784f2c8544dba86505c86f P 84332eccbe6642c886c8ae3489db3b1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:06:52.435019 2703 raft_consensus.cc:2258] T 0a68b775e3784f2c8544dba86505c86f P 84332eccbe6642c886c8ae3489db3b1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:06:52.435019 2704 raft_consensus.cc:2258] T 0a68b775e3784f2c8544dba86505c86f P a2564232d24e4d959fa3c573a3b9deb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:06:52.435088 2706 tablet_replica.cc:330] T 0a68b775e3784f2c8544dba86505c86f P 66516f726b524477910e30b8165b19f2: stopping tablet replica
I20260620 00:06:52.435135 2706 raft_consensus.cc:2229] T 0a68b775e3784f2c8544dba86505c86f P 66516f726b524477910e30b8165b19f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:06:52.435171 2706 raft_consensus.cc:2258] T 0a68b775e3784f2c8544dba86505c86f P 66516f726b524477910e30b8165b19f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:06:52.435326 2704 ts_tablet_manager.cc:1905] T 0a68b775e3784f2c8544dba86505c86f P a2564232d24e4d959fa3c573a3b9deb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:06:52.435595 2706 ts_tablet_manager.cc:1905] T 0a68b775e3784f2c8544dba86505c86f P 66516f726b524477910e30b8165b19f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:06:52.436945 2704 ts_tablet_manager.cc:1918] T 0a68b775e3784f2c8544dba86505c86f P a2564232d24e4d959fa3c573a3b9deb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 00:06:52.437016 2706 ts_tablet_manager.cc:1918] T 0a68b775e3784f2c8544dba86505c86f P 66516f726b524477910e30b8165b19f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 00:06:52.437050 2704 log.cc:1196] T 0a68b775e3784f2c8544dba86505c86f P a2564232d24e4d959fa3c573a3b9deb1: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-0/wal/wals/0a68b775e3784f2c8544dba86505c86f
I20260620 00:06:52.437065 2706 log.cc:1196] T 0a68b775e3784f2c8544dba86505c86f P 66516f726b524477910e30b8165b19f2: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-2/wal/wals/0a68b775e3784f2c8544dba86505c86f
I20260620 00:06:52.437287 2704 ts_tablet_manager.cc:1939] T 0a68b775e3784f2c8544dba86505c86f P a2564232d24e4d959fa3c573a3b9deb1: Deleting consensus metadata
I20260620 00:06:52.437299 2706 ts_tablet_manager.cc:1939] T 0a68b775e3784f2c8544dba86505c86f P 66516f726b524477910e30b8165b19f2: Deleting consensus metadata
I20260620 00:06:52.437698 2105 catalog_manager.cc:4841] TS a2564232d24e4d959fa3c573a3b9deb1 (127.0.98.1:37291): tablet 0a68b775e3784f2c8544dba86505c86f (table test-table [id=aec9e39fc81c418ebdbf6884f9c2c2fb]) successfully deleted
I20260620 00:06:52.437749 2703 ts_tablet_manager.cc:1905] T 0a68b775e3784f2c8544dba86505c86f P 84332eccbe6642c886c8ae3489db3b1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:06:52.437887 2119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55552:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 00:06:52.438001 2105 catalog_manager.cc:4841] TS 66516f726b524477910e30b8165b19f2 (127.0.98.3:38535): tablet 0a68b775e3784f2c8544dba86505c86f (table test-table [id=aec9e39fc81c418ebdbf6884f9c2c2fb]) successfully deleted
W20260620 00:06:52.438048 2119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:06:52.439245 2703 ts_tablet_manager.cc:1918] T 0a68b775e3784f2c8544dba86505c86f P 84332eccbe6642c886c8ae3489db3b1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 00:06:52.439278 2537 tablet_service.cc:1467] Processing CreateTablet for tablet 1e9175a5a58d4e8ebb147a0de1dfb4ad (DEFAULT_TABLE table=test-table [id=c086b86fcdf240a0a4946a5341424a90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:06:52.439278 2397 tablet_service.cc:1467] Processing CreateTablet for tablet 1e9175a5a58d4e8ebb147a0de1dfb4ad (DEFAULT_TABLE table=test-table [id=c086b86fcdf240a0a4946a5341424a90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:06:52.439385 2537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e9175a5a58d4e8ebb147a0de1dfb4ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:06:52.439385 2397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e9175a5a58d4e8ebb147a0de1dfb4ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:06:52.440747 2714 tablet_bootstrap.cc:492] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 84332eccbe6642c886c8ae3489db3b1b: Bootstrap starting.
I20260620 00:06:52.440766 2715 tablet_bootstrap.cc:492] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 66516f726b524477910e30b8165b19f2: Bootstrap starting.
I20260620 00:06:52.441057 2714 tablet_bootstrap.cc:654] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 84332eccbe6642c886c8ae3489db3b1b: Neither blocks nor log segments found. Creating new log.
I20260620 00:06:52.441109 2715 tablet_bootstrap.cc:654] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 66516f726b524477910e30b8165b19f2: Neither blocks nor log segments found. Creating new log.
I20260620 00:06:52.439308 2703 log.cc:1196] T 0a68b775e3784f2c8544dba86505c86f P 84332eccbe6642c886c8ae3489db3b1b: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-1/wal/wals/0a68b775e3784f2c8544dba86505c86f
I20260620 00:06:52.441507 2257 tablet_service.cc:1467] Processing CreateTablet for tablet 1e9175a5a58d4e8ebb147a0de1dfb4ad (DEFAULT_TABLE table=test-table [id=c086b86fcdf240a0a4946a5341424a90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:06:52.441613 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e9175a5a58d4e8ebb147a0de1dfb4ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:06:52.441612 2703 ts_tablet_manager.cc:1939] T 0a68b775e3784f2c8544dba86505c86f P 84332eccbe6642c886c8ae3489db3b1b: Deleting consensus metadata
I20260620 00:06:52.441772 2715 tablet_bootstrap.cc:492] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 66516f726b524477910e30b8165b19f2: No bootstrap required, opened a new log
I20260620 00:06:52.441813 2715 ts_tablet_manager.cc:1397] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 66516f726b524477910e30b8165b19f2: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 00:06:52.441937 2105 catalog_manager.cc:4841] TS 84332eccbe6642c886c8ae3489db3b1b (127.0.98.2:36571): tablet 0a68b775e3784f2c8544dba86505c86f (table test-table [id=aec9e39fc81c418ebdbf6884f9c2c2fb]) successfully deleted
I20260620 00:06:52.441938 2715 raft_consensus.cc:348] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 66516f726b524477910e30b8165b19f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } }
I20260620 00:06:52.442001 2715 raft_consensus.cc:374] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 66516f726b524477910e30b8165b19f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:06:52.442021 2715 raft_consensus.cc:729] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 66516f726b524477910e30b8165b19f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66516f726b524477910e30b8165b19f2, State: Initialized, Role: FOLLOWER
I20260620 00:06:52.442065 2715 consensus_queue.cc:260] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 66516f726b524477910e30b8165b19f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } }
I20260620 00:06:52.442147 2715 ts_tablet_manager.cc:1428] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 66516f726b524477910e30b8165b19f2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:52.442966 2719 tablet_bootstrap.cc:492] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P a2564232d24e4d959fa3c573a3b9deb1: Bootstrap starting.
I20260620 00:06:52.443332 2719 tablet_bootstrap.cc:654] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P a2564232d24e4d959fa3c573a3b9deb1: Neither blocks nor log segments found. Creating new log.
I20260620 00:06:52.443400 2714 tablet_bootstrap.cc:492] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 84332eccbe6642c886c8ae3489db3b1b: No bootstrap required, opened a new log
I20260620 00:06:52.443446 2714 ts_tablet_manager.cc:1397] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 84332eccbe6642c886c8ae3489db3b1b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 00:06:52.443570 2714 raft_consensus.cc:348] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 84332eccbe6642c886c8ae3489db3b1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } }
I20260620 00:06:52.443642 2714 raft_consensus.cc:374] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 84332eccbe6642c886c8ae3489db3b1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:06:52.443666 2714 raft_consensus.cc:729] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 84332eccbe6642c886c8ae3489db3b1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84332eccbe6642c886c8ae3489db3b1b, State: Initialized, Role: FOLLOWER
I20260620 00:06:52.443711 2714 consensus_queue.cc:260] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 84332eccbe6642c886c8ae3489db3b1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } }
I20260620 00:06:52.443791 2714 ts_tablet_manager.cc:1428] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 84332eccbe6642c886c8ae3489db3b1b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 00:06:52.443962 2719 tablet_bootstrap.cc:492] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P a2564232d24e4d959fa3c573a3b9deb1: No bootstrap required, opened a new log
I20260620 00:06:52.444006 2719 ts_tablet_manager.cc:1397] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P a2564232d24e4d959fa3c573a3b9deb1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:06:52.444111 2719 raft_consensus.cc:348] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P a2564232d24e4d959fa3c573a3b9deb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } }
I20260620 00:06:52.444178 2719 raft_consensus.cc:374] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P a2564232d24e4d959fa3c573a3b9deb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:06:52.444198 2719 raft_consensus.cc:729] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P a2564232d24e4d959fa3c573a3b9deb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2564232d24e4d959fa3c573a3b9deb1, State: Initialized, Role: FOLLOWER
I20260620 00:06:52.444237 2719 consensus_queue.cc:260] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P a2564232d24e4d959fa3c573a3b9deb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } }
I20260620 00:06:52.444332 2719 ts_tablet_manager.cc:1428] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P a2564232d24e4d959fa3c573a3b9deb1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:52.713716 2698 raft_consensus.cc:482] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 84332eccbe6642c886c8ae3489db3b1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:06:52.713829 2698 raft_consensus.cc:504] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 84332eccbe6642c886c8ae3489db3b1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } }
I20260620 00:06:52.714010 2698 leader_election.cc:288] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 84332eccbe6642c886c8ae3489db3b1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2564232d24e4d959fa3c573a3b9deb1 (127.0.98.1:37291), 66516f726b524477910e30b8165b19f2 (127.0.98.3:38535)
I20260620 00:06:52.714278 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e9175a5a58d4e8ebb147a0de1dfb4ad" candidate_uuid: "84332eccbe6642c886c8ae3489db3b1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2564232d24e4d959fa3c573a3b9deb1" is_pre_election: true
I20260620 00:06:52.714301 2557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e9175a5a58d4e8ebb147a0de1dfb4ad" candidate_uuid: "84332eccbe6642c886c8ae3489db3b1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66516f726b524477910e30b8165b19f2" is_pre_election: true
I20260620 00:06:52.714391 2277 raft_consensus.cc:2454] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P a2564232d24e4d959fa3c573a3b9deb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84332eccbe6642c886c8ae3489db3b1b in term 0.
I20260620 00:06:52.714387 2557 raft_consensus.cc:2454] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 66516f726b524477910e30b8165b19f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84332eccbe6642c886c8ae3489db3b1b in term 0.
I20260620 00:06:52.714596 2349 leader_election.cc:302] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 84332eccbe6642c886c8ae3489db3b1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66516f726b524477910e30b8165b19f2, 84332eccbe6642c886c8ae3489db3b1b; no voters:
I20260620 00:06:52.714702 2698 raft_consensus.cc:2781] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 84332eccbe6642c886c8ae3489db3b1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:06:52.714752 2698 raft_consensus.cc:482] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 84332eccbe6642c886c8ae3489db3b1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:06:52.714787 2698 raft_consensus.cc:3037] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 84332eccbe6642c886c8ae3489db3b1b [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:06:52.715526 2698 raft_consensus.cc:504] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 84332eccbe6642c886c8ae3489db3b1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } }
I20260620 00:06:52.715692 2698 leader_election.cc:288] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 84332eccbe6642c886c8ae3489db3b1b [CANDIDATE]: Term 1 election: Requested vote from peers a2564232d24e4d959fa3c573a3b9deb1 (127.0.98.1:37291), 66516f726b524477910e30b8165b19f2 (127.0.98.3:38535)
I20260620 00:06:52.715888 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e9175a5a58d4e8ebb147a0de1dfb4ad" candidate_uuid: "84332eccbe6642c886c8ae3489db3b1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2564232d24e4d959fa3c573a3b9deb1"
I20260620 00:06:52.715945 2557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e9175a5a58d4e8ebb147a0de1dfb4ad" candidate_uuid: "84332eccbe6642c886c8ae3489db3b1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66516f726b524477910e30b8165b19f2"
I20260620 00:06:52.715994 2277 raft_consensus.cc:3037] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P a2564232d24e4d959fa3c573a3b9deb1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:06:52.716037 2557 raft_consensus.cc:3037] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 66516f726b524477910e30b8165b19f2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:06:52.716671 2277 raft_consensus.cc:2454] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P a2564232d24e4d959fa3c573a3b9deb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84332eccbe6642c886c8ae3489db3b1b in term 1.
I20260620 00:06:52.716671 2557 raft_consensus.cc:2454] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 66516f726b524477910e30b8165b19f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84332eccbe6642c886c8ae3489db3b1b in term 1.
I20260620 00:06:52.716925 2349 leader_election.cc:302] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 84332eccbe6642c886c8ae3489db3b1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84332eccbe6642c886c8ae3489db3b1b, a2564232d24e4d959fa3c573a3b9deb1; no voters:
I20260620 00:06:52.717061 2698 raft_consensus.cc:2781] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 84332eccbe6642c886c8ae3489db3b1b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:06:52.717226 2698 raft_consensus.cc:686] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 84332eccbe6642c886c8ae3489db3b1b [term 1 LEADER]: Becoming Leader. State: Replica: 84332eccbe6642c886c8ae3489db3b1b, State: Running, Role: LEADER
I20260620 00:06:52.717293 2698 consensus_queue.cc:237] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 84332eccbe6642c886c8ae3489db3b1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } }
I20260620 00:06:52.717994 2119 catalog_manager.cc:5495] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 84332eccbe6642c886c8ae3489db3b1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 84332eccbe6642c886c8ae3489db3b1b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "84332eccbe6642c886c8ae3489db3b1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84332eccbe6642c886c8ae3489db3b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 } health_report { overall_health: UNKNOWN } } }
I20260620 00:06:52.730346 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:52.727940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39792 (local address 127.0.98.2:36571)
0620 00:06:52.728066 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:52.728068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:52.728076 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:06:52.728110 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:52.728110 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:52.728119 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:52.728187 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:06:52.728292 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:52.728757 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:52.729620 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:52.729877 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:52.729934 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:52.730150 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:06:52.730176 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:06:52.730211 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:52.730269 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 00:06:52.730284 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":83,"threads_started":1}
I20260620 00:06:52.732718 2277 raft_consensus.cc:1261] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P a2564232d24e4d959fa3c573a3b9deb1 [term 1 FOLLOWER]: Refusing update from remote peer 84332eccbe6642c886c8ae3489db3b1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:06:52.732895 2557 raft_consensus.cc:1261] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 66516f726b524477910e30b8165b19f2 [term 1 FOLLOWER]: Refusing update from remote peer 84332eccbe6642c886c8ae3489db3b1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:06:52.732963 2733 consensus_queue.cc:1035] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 84332eccbe6642c886c8ae3489db3b1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2564232d24e4d959fa3c573a3b9deb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:06:52.733142 2733 consensus_queue.cc:1035] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 84332eccbe6642c886c8ae3489db3b1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "66516f726b524477910e30b8165b19f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 00:06:52.737622 2687 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 00:06:54.259188 2120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:06:54.259303 2120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:06:54.260165 2120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 49f4add4939e45c4998254385ad73d44: Sending DeleteTablet for 3 replicas of tablet 1e9175a5a58d4e8ebb147a0de1dfb4ad
I20260620 00:06:54.260461 2397 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e9175a5a58d4e8ebb147a0de1dfb4ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:06:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39774
I20260620 00:06:54.260469 2257 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e9175a5a58d4e8ebb147a0de1dfb4ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:06:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38754
I20260620 00:06:54.260517 2537 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e9175a5a58d4e8ebb147a0de1dfb4ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:06:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49828
I20260620 00:06:54.260548 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 2192
I20260620 00:06:54.261059 2762 tablet_replica.cc:330] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 66516f726b524477910e30b8165b19f2: stopping tablet replica
I20260620 00:06:54.261183 2762 raft_consensus.cc:2229] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 66516f726b524477910e30b8165b19f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:06:54.261229 2762 raft_consensus.cc:2258] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 66516f726b524477910e30b8165b19f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:06:54.261255 2761 tablet_replica.cc:330] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 84332eccbe6642c886c8ae3489db3b1b: stopping tablet replica
I20260620 00:06:54.261309 2761 raft_consensus.cc:2229] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 84332eccbe6642c886c8ae3489db3b1b [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:06:54.261400 2761 raft_consensus.cc:2258] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 84332eccbe6642c886c8ae3489db3b1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:06:54.261554 2762 ts_tablet_manager.cc:1905] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 66516f726b524477910e30b8165b19f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:06:54.261799 2761 ts_tablet_manager.cc:1905] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 84332eccbe6642c886c8ae3489db3b1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:06:54.263430 2761 ts_tablet_manager.cc:1918] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 84332eccbe6642c886c8ae3489db3b1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 00:06:54.263485 2761 log.cc:1196] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 84332eccbe6642c886c8ae3489db3b1b: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-1/wal/wals/1e9175a5a58d4e8ebb147a0de1dfb4ad
I20260620 00:06:54.263746 2761 ts_tablet_manager.cc:1939] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 84332eccbe6642c886c8ae3489db3b1b: Deleting consensus metadata
I20260620 00:06:54.263775 2762 ts_tablet_manager.cc:1918] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 66516f726b524477910e30b8165b19f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 00:06:54.263832 2762 log.cc:1196] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 66516f726b524477910e30b8165b19f2: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914001714546-392-0/minicluster-data/ts-2/wal/wals/1e9175a5a58d4e8ebb147a0de1dfb4ad
I20260620 00:06:54.264165 2762 ts_tablet_manager.cc:1939] T 1e9175a5a58d4e8ebb147a0de1dfb4ad P 66516f726b524477910e30b8165b19f2: Deleting consensus metadata
I20260620 00:06:54.264269 2105 catalog_manager.cc:4841] TS 84332eccbe6642c886c8ae3489db3b1b (127.0.98.2:36571): tablet 1e9175a5a58d4e8ebb147a0de1dfb4ad (table test-table [id=c086b86fcdf240a0a4946a5341424a90]) successfully deleted
I20260620 00:06:54.264484 2105 catalog_manager.cc:4841] TS 66516f726b524477910e30b8165b19f2 (127.0.98.3:38535): tablet 1e9175a5a58d4e8ebb147a0de1dfb4ad (table test-table [id=c086b86fcdf240a0a4946a5341424a90]) successfully deleted
W20260620 00:06:54.268342 2105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37291 (error 108)
I20260620 00:06:54.268471 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 2332
I20260620 00:06:54.270162 2764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:54.269620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37291 (local address 127.0.0.1:38784)
0620 00:06:54.270073 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:54.270124 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":66,"threads_started":1}
W20260620 00:06:54.270345 2105 catalog_manager.cc:4584] TS a2564232d24e4d959fa3c573a3b9deb1 (127.0.98.1:37291): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1e9175a5a58d4e8ebb147a0de1dfb4ad: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37291: connect: Connection refused (error 111)
I20260620 00:06:54.274356 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 2472
I20260620 00:06:54.279134 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 2088
2026-06-20T00:06:54Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6018 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:06:54 dist-test-slave-grqt krb5kdc[2767](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:06:54 dist-test-slave-grqt krb5kdc[2767](info): set up 2 sockets
Jun 20 00:06:54 dist-test-slave-grqt krb5kdc[2767](info): commencing operation
krb5kdc: starting...
W20260620 00:06:55.329489 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:06:55 dist-test-slave-grqt krb5kdc[2767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:06:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:06:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:06:55.453054 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42833
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42833
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:06:55.535403 2780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:06:55.535668 2780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:06:55.535706 2780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:06:55.536873 2780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:06:55.536940 2780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:06:55.536966 2780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:06:55.536996 2780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:06:55.537021 2780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:06:55.538659 2780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:06:55.538940 2780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:06:55.539170 2780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:06:55.544437 2785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:06:55.544442 2780 server_base.cc:1034] running on GCE node
W20260620 00:06:55.544612 2788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:06:55.544433 2786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:06:55.544997 2780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:06:55.545295 2780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:06:55.546474 2780 hybrid_clock.cc:648] HybridClock initialized: now 1781914015546452 us; error 44 us; skew 500 ppm
Jun 20 00:06:55 dist-test-slave-grqt krb5kdc[2767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914015, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:06:55.548981 2780 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:06:55.549522 2780 webserver.cc:460] Webserver started at http://127.0.98.62:35511/ using document root <none> and password file <none>
I20260620 00:06:55.549779 2780 fs_manager.cc:359] Metadata directory not provided
I20260620 00:06:55.549849 2780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:06:55.550007 2780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:06:55.551090 2780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "d5eb5cb8d4b14e21b76f2faed71ac78a"
format_stamp: "Formatted at 2026-06-20 00:06:55 on dist-test-slave-grqt"
server_key: "e51cffb850c20b6181cc1b447c7a19c7"
server_key_iv: "769644a63b7bee7ff9f1b2672d8a410f"
server_key_version: "encryptionkey@0"
I20260620 00:06:55.551465 2780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "d5eb5cb8d4b14e21b76f2faed71ac78a"
format_stamp: "Formatted at 2026-06-20 00:06:55 on dist-test-slave-grqt"
server_key: "e51cffb850c20b6181cc1b447c7a19c7"
server_key_iv: "769644a63b7bee7ff9f1b2672d8a410f"
server_key_version: "encryptionkey@0"
I20260620 00:06:55.552871 2780 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:06:55.553788 2795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:55.553994 2780 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:06:55.554101 2780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "d5eb5cb8d4b14e21b76f2faed71ac78a"
format_stamp: "Formatted at 2026-06-20 00:06:55 on dist-test-slave-grqt"
server_key: "e51cffb850c20b6181cc1b447c7a19c7"
server_key_iv: "769644a63b7bee7ff9f1b2672d8a410f"
server_key_version: "encryptionkey@0"
I20260620 00:06:55.554193 2780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:06:55.578454 2780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:06:55.581032 2780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:06:55.581213 2780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:06:55.585348 2780 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42833
I20260620 00:06:55.585457 2847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42833 every 8 connection(s)
I20260620 00:06:55.585974 2780 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:06:55.586673 2848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:06:55.588073 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 2780
I20260620 00:06:55.588168 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:06:55.588353 392 external_mini_cluster.cc:1428] Setting key cf36d5927ae8214babe6316e565033ed
I20260620 00:06:55.589255 2848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5eb5cb8d4b14e21b76f2faed71ac78a: Bootstrap starting.
I20260620 00:06:55.589879 2848 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5eb5cb8d4b14e21b76f2faed71ac78a: Neither blocks nor log segments found. Creating new log.
I20260620 00:06:55.590188 2848 log.cc:824] T 00000000000000000000000000000000 P d5eb5cb8d4b14e21b76f2faed71ac78a: Log is configured to *not* fsync() on all Append() calls
I20260620 00:06:55.590871 2848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5eb5cb8d4b14e21b76f2faed71ac78a: No bootstrap required, opened a new log
I20260620 00:06:55.592664 2848 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5eb5cb8d4b14e21b76f2faed71ac78a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5eb5cb8d4b14e21b76f2faed71ac78a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42833 } }
I20260620 00:06:55.592844 2848 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5eb5cb8d4b14e21b76f2faed71ac78a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:06:55.592890 2848 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5eb5cb8d4b14e21b76f2faed71ac78a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5eb5cb8d4b14e21b76f2faed71ac78a, State: Initialized, Role: FOLLOWER
I20260620 00:06:55.593039 2848 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5eb5cb8d4b14e21b76f2faed71ac78a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5eb5cb8d4b14e21b76f2faed71ac78a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42833 } }
I20260620 00:06:55.593133 2848 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5eb5cb8d4b14e21b76f2faed71ac78a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:06:55.593176 2848 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5eb5cb8d4b14e21b76f2faed71ac78a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:06:55.593238 2848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5eb5cb8d4b14e21b76f2faed71ac78a [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:06:55.593828 2848 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5eb5cb8d4b14e21b76f2faed71ac78a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5eb5cb8d4b14e21b76f2faed71ac78a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42833 } }
I20260620 00:06:55.593950 2848 leader_election.cc:302] T 00000000000000000000000000000000 P d5eb5cb8d4b14e21b76f2faed71ac78a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5eb5cb8d4b14e21b76f2faed71ac78a; no voters:
I20260620 00:06:55.594125 2848 leader_election.cc:288] T 00000000000000000000000000000000 P d5eb5cb8d4b14e21b76f2faed71ac78a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:06:55.594199 2853 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5eb5cb8d4b14e21b76f2faed71ac78a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:06:55.594363 2853 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5eb5cb8d4b14e21b76f2faed71ac78a [term 1 LEADER]: Becoming Leader. State: Replica: d5eb5cb8d4b14e21b76f2faed71ac78a, State: Running, Role: LEADER
I20260620 00:06:55.594403 2848 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5eb5cb8d4b14e21b76f2faed71ac78a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:06:55.594461 2853 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5eb5cb8d4b14e21b76f2faed71ac78a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5eb5cb8d4b14e21b76f2faed71ac78a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42833 } }
I20260620 00:06:55.594812 2854 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5eb5cb8d4b14e21b76f2faed71ac78a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5eb5cb8d4b14e21b76f2faed71ac78a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5eb5cb8d4b14e21b76f2faed71ac78a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42833 } } }
I20260620 00:06:55.594936 2854 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5eb5cb8d4b14e21b76f2faed71ac78a [sys.catalog]: This master's current role is: LEADER
I20260620 00:06:55.595129 2860 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 20 00:06:55 dist-test-slave-grqt krb5kdc[2767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:06:55.595674 2860 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:06:55.594796 2853 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5eb5cb8d4b14e21b76f2faed71ac78a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5eb5cb8d4b14e21b76f2faed71ac78a. Latest consensus state: current_term: 1 leader_uuid: "d5eb5cb8d4b14e21b76f2faed71ac78a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5eb5cb8d4b14e21b76f2faed71ac78a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42833 } } }
I20260620 00:06:55.595841 2853 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5eb5cb8d4b14e21b76f2faed71ac78a [sys.catalog]: This master's current role is: LEADER
I20260620 00:06:55.597890 2860 catalog_manager.cc:1347] Generated new cluster ID: 7232c43b13f1480498e797c2cbec0c57
I20260620 00:06:55.597944 2860 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:06:55.600096 2851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:55.588937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36484 (local address 127.0.98.62:42833)
0620 00:06:55.589152 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:55.589160 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:55.589191 (+ 31us) server_negotiation.cc:407] Connection header received
0620 00:06:55.589683 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:55.589690 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:55.589741 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:55.589900 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:55.590605 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:55.591571 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:55.592336 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:55.592754 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:55.592909 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:55.595919 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:55.595924 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:55.595924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:55.595936 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:55.598447 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:55.598922 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:55.598926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:55.598926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:55.599037 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:55.599262 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:55.599263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:55.599263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:55.599501 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:55.599661 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:55.599769 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 00:06:55.599911 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":83,"threads_started":1}
I20260620 00:06:55.609169 2860 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:06:55.609834 2860 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:06:55.624830 2860 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5eb5cb8d4b14e21b76f2faed71ac78a: Generated new TSK 0
I20260620 00:06:55.625137 2860 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:06:55.660215 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42833
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:06:55.768177 2874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:06:55.768396 2874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:06:55.768421 2874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:06:55.769665 2874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:06:55.769716 2874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:06:55.769745 2874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:06:55.771281 2874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:06:55.771484 2874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:06:55.771692 2874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:06:55.777280 2879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:06:55.777457 2882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:06:55.777415 2874 server_base.cc:1034] running on GCE node
W20260620 00:06:55.777518 2880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:06:55.777822 2874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:06:55.778084 2874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:06:55.779263 2874 hybrid_clock.cc:648] HybridClock initialized: now 1781914015779248 us; error 30 us; skew 500 ppm
Jun 20 00:06:55 dist-test-slave-grqt krb5kdc[2767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914015, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:06:55.782124 2874 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:06:55.782586 2874 webserver.cc:460] Webserver started at http://127.0.98.1:37799/ using document root <none> and password file <none>
I20260620 00:06:55.782833 2874 fs_manager.cc:359] Metadata directory not provided
I20260620 00:06:55.782881 2874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:06:55.783052 2874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:06:55.784197 2874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-0/data/instance:
uuid: "c86513a364314f9a93296f681c71577b"
format_stamp: "Formatted at 2026-06-20 00:06:55 on dist-test-slave-grqt"
server_key: "2600b88abde37036a38f7aec8515f5b8"
server_key_iv: "786114bcf536dce4c4ee25aa1a778aec"
server_key_version: "encryptionkey@0"
I20260620 00:06:55.784574 2874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c86513a364314f9a93296f681c71577b"
format_stamp: "Formatted at 2026-06-20 00:06:55 on dist-test-slave-grqt"
server_key: "2600b88abde37036a38f7aec8515f5b8"
server_key_iv: "786114bcf536dce4c4ee25aa1a778aec"
server_key_version: "encryptionkey@0"
I20260620 00:06:55.786093 2874 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:06:55.787002 2889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:55.787334 2874 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:06:55.787420 2874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-0/wal
uuid: "c86513a364314f9a93296f681c71577b"
format_stamp: "Formatted at 2026-06-20 00:06:55 on dist-test-slave-grqt"
server_key: "2600b88abde37036a38f7aec8515f5b8"
server_key_iv: "786114bcf536dce4c4ee25aa1a778aec"
server_key_version: "encryptionkey@0"
I20260620 00:06:55.787518 2874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:06:55.803207 2874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:06:55.805720 2874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:06:55.805896 2874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:06:55.806252 2874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:06:55.806588 2874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:06:55.806625 2874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:55.806649 2874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:06:55.806695 2874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:55.812593 2874 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33049
I20260620 00:06:55.812642 3002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33049 every 8 connection(s)
I20260620 00:06:55.812969 2874 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:06:55.816491 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 2874
I20260620 00:06:55.816594 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:06:55.816797 392 external_mini_cluster.cc:1428] Setting key 0c2a92a097c95a1c89a550c6af3fdf92
Jun 20 00:06:55 dist-test-slave-grqt krb5kdc[2767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914015, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:06:55.824002 2851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:55.813763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34017 (local address 127.0.98.62:42833)
0620 00:06:55.813849 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:55.813851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:55.814057 (+ 206us) server_negotiation.cc:407] Connection header received
0620 00:06:55.815025 (+ 968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:55.815026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:55.815038 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:55.815107 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:55.815848 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:55.816477 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:55.817771 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:55.818114 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:55.818189 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:55.820709 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:55.820710 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:55.820711 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:55.820719 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:55.822460 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:55.822900 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:55.822901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:55.822901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:55.822972 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:55.823215 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:55.823216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:55.823216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:55.823369 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:55.823453 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:55.823770 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 00:06:55.823870 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 00:06:55.824112 3005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:55.813731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42833 (local address 127.0.98.1:34017)
0620 00:06:55.813916 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:55.813962 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:55.814896 (+ 934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:55.815248 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:55.815253 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:55.815537 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:06:55.815756 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:55.815761 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:55.816617 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:55.816618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:55.817680 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:55.817683 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:55.818202 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:55.818203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:55.818313 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:55.818958 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:06:55.818970 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:06:55.820611 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:06:55.822556 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:55.822563 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:55.822565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:55.822822 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:55.823046 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:55.823046 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:55.823046 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:55.823151 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:55.823480 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:06:55.823486 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:06:55.823620 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 00:06:55.823720 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 00:06:55.823881 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":91,"threads_started":1}
I20260620 00:06:55.824790 3003 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42833
I20260620 00:06:55.824911 3003 heartbeater.cc:464] Registering TS with master...
I20260620 00:06:55.825129 3003 heartbeater.cc:511] Master 127.0.98.62:42833 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 00:06:55.825788 2812 ts_manager.cc:194] Registered new tserver with Master: c86513a364314f9a93296f681c71577b (127.0.98.1:33049)
I20260620 00:06:55.826439 2812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34017
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:06:55.845587 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42833
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:06:55.928735 3008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:06:55.928956 3008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:06:55.928982 3008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:06:55.930251 3008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:06:55.930297 3008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:06:55.930325 3008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:06:55.932410 3008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:06:55.932713 3008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:06:55.932973 3008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:06:55.938889 3014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:06:55.938884 3016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:06:55.938980 3013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:06:55.939261 3008 server_base.cc:1034] running on GCE node
I20260620 00:06:55.939581 3008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:06:55.939905 3008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:06:55.941095 3008 hybrid_clock.cc:648] HybridClock initialized: now 1781914015941069 us; error 38 us; skew 500 ppm
Jun 20 00:06:55 dist-test-slave-grqt krb5kdc[2767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914015, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:06:55.944175 3008 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:06:55.944751 3008 webserver.cc:460] Webserver started at http://127.0.98.2:42989/ using document root <none> and password file <none>
I20260620 00:06:55.945017 3008 fs_manager.cc:359] Metadata directory not provided
I20260620 00:06:55.945087 3008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:06:55.945238 3008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:06:55.946425 3008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-1/data/instance:
uuid: "b80d0a4154914fbc9e6afa08d4a18a63"
format_stamp: "Formatted at 2026-06-20 00:06:55 on dist-test-slave-grqt"
server_key: "a0fbd1a9494ddd60176e6746d02dbc1c"
server_key_iv: "51b6f68469103d2de28966955903d751"
server_key_version: "encryptionkey@0"
I20260620 00:06:55.946810 3008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b80d0a4154914fbc9e6afa08d4a18a63"
format_stamp: "Formatted at 2026-06-20 00:06:55 on dist-test-slave-grqt"
server_key: "a0fbd1a9494ddd60176e6746d02dbc1c"
server_key_iv: "51b6f68469103d2de28966955903d751"
server_key_version: "encryptionkey@0"
I20260620 00:06:55.948311 3008 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:06:55.949198 3023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:55.949441 3008 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:06:55.949551 3008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-1/wal
uuid: "b80d0a4154914fbc9e6afa08d4a18a63"
format_stamp: "Formatted at 2026-06-20 00:06:55 on dist-test-slave-grqt"
server_key: "a0fbd1a9494ddd60176e6746d02dbc1c"
server_key_iv: "51b6f68469103d2de28966955903d751"
server_key_version: "encryptionkey@0"
I20260620 00:06:55.949640 3008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:06:55.968612 3008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:06:55.971082 3008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:06:55.971285 3008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:06:55.971539 3008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:06:55.971856 3008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:06:55.971910 3008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:55.971952 3008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:06:55.971979 3008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:55.977819 3008 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44065
I20260620 00:06:55.977861 3136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44065 every 8 connection(s)
I20260620 00:06:55.978225 3008 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:06:55.980816 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 3008
I20260620 00:06:55.980909 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:06:55.981128 392 external_mini_cluster.cc:1428] Setting key 8ad1fb836367f74a3d444d6cfa079636
Jun 20 00:06:55 dist-test-slave-grqt krb5kdc[2767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914015, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:06:55.989316 2851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:55.979019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50051 (local address 127.0.98.62:42833)
0620 00:06:55.979084 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:55.979086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:55.979318 (+ 232us) server_negotiation.cc:407] Connection header received
0620 00:06:55.980474 (+ 1156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:55.980476 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:55.980489 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:55.980570 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:55.981555 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:55.982226 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:55.983285 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:55.983497 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:55.983568 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:55.986246 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:55.986247 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:55.986247 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:55.986255 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:55.987702 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:55.988241 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:55.988242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:55.988242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:55.988300 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:55.988584 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:55.988585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:55.988585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:55.988691 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:55.988741 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:55.989151 (+ 410us) server_negotiation.cc:299] Negotiation successful
0620 00:06:55.989247 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 00:06:55.989606 3139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:55.979003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42833 (local address 127.0.98.2:50051)
0620 00:06:55.979185 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:55.979228 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:55.980329 (+ 1101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:55.980657 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:55.980662 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:55.981122 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:06:55.981433 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:55.981440 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:55.982329 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:55.982330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:55.983180 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:55.983182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:55.983588 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:55.983589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:55.983727 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:55.984330 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:06:55.984338 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:06:55.986140 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:06:55.987793 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:55.987801 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:55.987804 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:55.988159 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:55.988372 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:55.988373 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:55.988374 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:55.988512 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:55.988760 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:06:55.988766 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:06:55.988955 (+ 189us) client_negotiation.cc:769] Sending connection context
0620 00:06:55.989089 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 00:06:55.989319 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 00:06:55.990173 3137 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42833
I20260620 00:06:55.990312 3137 heartbeater.cc:464] Registering TS with master...
I20260620 00:06:55.990568 3137 heartbeater.cc:511] Master 127.0.98.62:42833 requested a full tablet report, sending...
I20260620 00:06:55.991071 2812 ts_manager.cc:194] Registered new tserver with Master: b80d0a4154914fbc9e6afa08d4a18a63 (127.0.98.2:44065)
I20260620 00:06:55.991504 2812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50051
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:06:56.012353 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42833
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:06:56.095723 3142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:06:56.095994 3142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:06:56.096025 3142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:06:56.097280 3142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:06:56.097325 3142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:06:56.097354 3142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:06:56.098997 3142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:06:56.099275 3142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:06:56.099525 3142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:06:56.104790 3148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:06:56.104791 3150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:06:56.104791 3147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:06:56.104857 3142 server_base.cc:1034] running on GCE node
I20260620 00:06:56.105283 3142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:06:56.105638 3142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:06:56.106827 3142 hybrid_clock.cc:648] HybridClock initialized: now 1781914016106806 us; error 35 us; skew 500 ppm
Jun 20 00:06:56 dist-test-slave-grqt krb5kdc[2767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914016, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:06:56.109567 3142 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:06:56.110074 3142 webserver.cc:460] Webserver started at http://127.0.98.3:44181/ using document root <none> and password file <none>
I20260620 00:06:56.110267 3142 fs_manager.cc:359] Metadata directory not provided
I20260620 00:06:56.110309 3142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:06:56.110402 3142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:06:56.111506 3142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-2/data/instance:
uuid: "25327a255f87464798e5acfff19547c7"
format_stamp: "Formatted at 2026-06-20 00:06:56 on dist-test-slave-grqt"
server_key: "e838ef31015cec0a45cacd1f59b3903b"
server_key_iv: "ebbc151217865c7c61454cde86c4c31f"
server_key_version: "encryptionkey@0"
I20260620 00:06:56.111831 3142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-2/wal/instance:
uuid: "25327a255f87464798e5acfff19547c7"
format_stamp: "Formatted at 2026-06-20 00:06:56 on dist-test-slave-grqt"
server_key: "e838ef31015cec0a45cacd1f59b3903b"
server_key_iv: "ebbc151217865c7c61454cde86c4c31f"
server_key_version: "encryptionkey@0"
I20260620 00:06:56.113189 3142 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:06:56.114042 3157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:56.114235 3142 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:06:56.114342 3142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-2/wal
uuid: "25327a255f87464798e5acfff19547c7"
format_stamp: "Formatted at 2026-06-20 00:06:56 on dist-test-slave-grqt"
server_key: "e838ef31015cec0a45cacd1f59b3903b"
server_key_iv: "ebbc151217865c7c61454cde86c4c31f"
server_key_version: "encryptionkey@0"
I20260620 00:06:56.114419 3142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:06:56.132983 3142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:06:56.135113 3142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:06:56.135262 3142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:06:56.135502 3142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:06:56.135816 3142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:06:56.135872 3142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:56.135914 3142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:06:56.135941 3142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:56.141739 3142 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36257
I20260620 00:06:56.141824 3270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36257 every 8 connection(s)
I20260620 00:06:56.142143 3142 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:06:56.147692 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 3142
I20260620 00:06:56.147811 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:06:56.147984 392 external_mini_cluster.cc:1428] Setting key c212c51b2b76c6206fe0e7357399ba11
Jun 20 00:06:56 dist-test-slave-grqt krb5kdc[2767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914016, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:06:56.151847 2851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:56.142924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48001 (local address 127.0.98.62:42833)
0620 00:06:56.142971 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:56.142973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:56.143173 (+ 200us) server_negotiation.cc:407] Connection header received
0620 00:06:56.144179 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:56.144180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:56.144192 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:56.144263 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:56.144926 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:56.145530 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:56.146511 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:56.146699 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:56.146749 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:56.149223 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:56.149224 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:56.149224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:56.149230 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:56.150524 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:56.150884 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:56.150885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:56.150885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:56.150937 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:56.151217 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:56.151218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:56.151218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:56.151361 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:56.151467 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:56.151704 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 00:06:56.151777 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 00:06:56.151996 3273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:56.142888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42833 (local address 127.0.98.3:48001)
0620 00:06:56.143052 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:56.143100 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:56.144041 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:56.144363 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:56.144367 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:56.144635 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:06:56.144827 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:56.144832 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:56.145624 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:56.145625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:56.146435 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:56.146437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:56.146770 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:56.146770 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:56.146867 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:56.147438 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:06:56.147447 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:06:56.149151 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:06:56.150599 (+ 1448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:56.150605 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:56.150606 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:56.150820 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:56.150993 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:56.150994 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:56.150994 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:56.151119 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:56.151434 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:06:56.151439 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:06:56.151558 (+ 119us) client_negotiation.cc:769] Sending connection context
0620 00:06:56.151659 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 00:06:56.151781 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":95,"threads_started":1}
I20260620 00:06:56.152534 3271 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42833
I20260620 00:06:56.152621 3271 heartbeater.cc:464] Registering TS with master...
I20260620 00:06:56.152822 3271 heartbeater.cc:511] Master 127.0.98.62:42833 requested a full tablet report, sending...
I20260620 00:06:56.153182 2812 ts_manager.cc:194] Registered new tserver with Master: 25327a255f87464798e5acfff19547c7 (127.0.98.3:36257)
I20260620 00:06:56.153720 2812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48001
I20260620 00:06:56.160063 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 00:06:56.172197 2851 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:06:56.170780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36494 (local address 127.0.98.62:42833)
0620 00:06:56.170837 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:56.170840 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:56.171083 (+ 243us) server_negotiation.cc:407] Connection header received
0620 00:06:56.171164 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:56.171166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:56.171176 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:56.171238 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:56.172120 (+ 882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36494: BlockingRecv error: recv got EOF from 127.0.0.1:36494 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 00:06:56.173642 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 2874
I20260620 00:06:56.179294 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 3008
I20260620 00:06:56.184217 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 3142
I20260620 00:06:56.188704 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 2780
2026-06-20T00:06:56Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1908 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:06:56 dist-test-slave-grqt krb5kdc[3283](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:06:56 dist-test-slave-grqt krb5kdc[3283](info): set up 2 sockets
Jun 20 00:06:56 dist-test-slave-grqt krb5kdc[3283](info): commencing operation
krb5kdc: starting...
W20260620 00:06:57.241179 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:06:57 dist-test-slave-grqt krb5kdc[3283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:06:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:06:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:06:57.360551 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33493
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:06:57.444410 3296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:06:57.444764 3296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:06:57.444806 3296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:06:57.446200 3296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:06:57.446285 3296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:06:57.446310 3296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:06:57.446332 3296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:06:57.446365 3296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:06:57.448192 3296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:06:57.448637 3296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:06:57.448966 3296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:06:57.454171 3302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:06:57.454146 3301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:06:57.454154 3304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:06:57.454684 3296 server_base.cc:1034] running on GCE node
I20260620 00:06:57.454978 3296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:06:57.455317 3296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:06:57.456521 3296 hybrid_clock.cc:648] HybridClock initialized: now 1781914017456495 us; error 45 us; skew 500 ppm
Jun 20 00:06:57 dist-test-slave-grqt krb5kdc[3283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914017, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:06:57.459208 3296 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:06:57.459685 3296 webserver.cc:460] Webserver started at http://127.0.98.62:43017/ using document root <none> and password file <none>
I20260620 00:06:57.459932 3296 fs_manager.cc:359] Metadata directory not provided
I20260620 00:06:57.460075 3296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:06:57.460229 3296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:06:57.461427 3296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "70e5675520ca4ef68f656ab79611adaa"
format_stamp: "Formatted at 2026-06-20 00:06:57 on dist-test-slave-grqt"
server_key: "8acd5c15bc4b8c35823a9896c0dab299"
server_key_iv: "aa4366d8990acbb863471401ba0a26f6"
server_key_version: "encryptionkey@0"
I20260620 00:06:57.461819 3296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "70e5675520ca4ef68f656ab79611adaa"
format_stamp: "Formatted at 2026-06-20 00:06:57 on dist-test-slave-grqt"
server_key: "8acd5c15bc4b8c35823a9896c0dab299"
server_key_iv: "aa4366d8990acbb863471401ba0a26f6"
server_key_version: "encryptionkey@0"
I20260620 00:06:57.463316 3296 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:06:57.464215 3311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:57.464507 3296 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:06:57.464618 3296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "70e5675520ca4ef68f656ab79611adaa"
format_stamp: "Formatted at 2026-06-20 00:06:57 on dist-test-slave-grqt"
server_key: "8acd5c15bc4b8c35823a9896c0dab299"
server_key_iv: "aa4366d8990acbb863471401ba0a26f6"
server_key_version: "encryptionkey@0"
I20260620 00:06:57.464718 3296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:06:57.477139 3296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:06:57.479660 3296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:06:57.479831 3296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:06:57.483851 3296 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33493
I20260620 00:06:57.483891 3363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33493 every 8 connection(s)
I20260620 00:06:57.484270 3296 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:06:57.485358 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 3296
I20260620 00:06:57.485460 3364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:06:57.485491 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:06:57.485713 392 external_mini_cluster.cc:1428] Setting key a0e7763f9661a61fa810b2bceaf098b3
I20260620 00:06:57.489099 3364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70e5675520ca4ef68f656ab79611adaa: Bootstrap starting.
I20260620 00:06:57.489884 3364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70e5675520ca4ef68f656ab79611adaa: Neither blocks nor log segments found. Creating new log.
I20260620 00:06:57.490185 3364 log.cc:824] T 00000000000000000000000000000000 P 70e5675520ca4ef68f656ab79611adaa: Log is configured to *not* fsync() on all Append() calls
I20260620 00:06:57.490890 3364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70e5675520ca4ef68f656ab79611adaa: No bootstrap required, opened a new log
I20260620 00:06:57.492645 3364 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70e5675520ca4ef68f656ab79611adaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e5675520ca4ef68f656ab79611adaa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33493 } }
I20260620 00:06:57.492813 3364 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70e5675520ca4ef68f656ab79611adaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:06:57.492862 3364 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70e5675520ca4ef68f656ab79611adaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70e5675520ca4ef68f656ab79611adaa, State: Initialized, Role: FOLLOWER
I20260620 00:06:57.493005 3364 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70e5675520ca4ef68f656ab79611adaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e5675520ca4ef68f656ab79611adaa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33493 } }
I20260620 00:06:57.493108 3364 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70e5675520ca4ef68f656ab79611adaa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:06:57.493153 3364 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70e5675520ca4ef68f656ab79611adaa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:06:57.493202 3364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70e5675520ca4ef68f656ab79611adaa [term 0 FOLLOWER]: Advancing to term 1
Jun 20 00:06:57 dist-test-slave-grqt krb5kdc[3283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:06:57.493872 3364 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70e5675520ca4ef68f656ab79611adaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e5675520ca4ef68f656ab79611adaa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33493 } }
I20260620 00:06:57.493984 3364 leader_election.cc:302] T 00000000000000000000000000000000 P 70e5675520ca4ef68f656ab79611adaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70e5675520ca4ef68f656ab79611adaa; no voters:
I20260620 00:06:57.494204 3364 leader_election.cc:288] T 00000000000000000000000000000000 P 70e5675520ca4ef68f656ab79611adaa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:06:57.494236 3369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70e5675520ca4ef68f656ab79611adaa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:06:57.494385 3369 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70e5675520ca4ef68f656ab79611adaa [term 1 LEADER]: Becoming Leader. State: Replica: 70e5675520ca4ef68f656ab79611adaa, State: Running, Role: LEADER
I20260620 00:06:57.494560 3364 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70e5675520ca4ef68f656ab79611adaa [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:06:57.494541 3369 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70e5675520ca4ef68f656ab79611adaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e5675520ca4ef68f656ab79611adaa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33493 } }
I20260620 00:06:57.495045 3371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70e5675520ca4ef68f656ab79611adaa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70e5675520ca4ef68f656ab79611adaa. Latest consensus state: current_term: 1 leader_uuid: "70e5675520ca4ef68f656ab79611adaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e5675520ca4ef68f656ab79611adaa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33493 } } }
I20260620 00:06:57.495142 3371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70e5675520ca4ef68f656ab79611adaa [sys.catalog]: This master's current role is: LEADER
I20260620 00:06:57.495378 3370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70e5675520ca4ef68f656ab79611adaa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70e5675520ca4ef68f656ab79611adaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e5675520ca4ef68f656ab79611adaa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33493 } } }
I20260620 00:06:57.495489 3370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70e5675520ca4ef68f656ab79611adaa [sys.catalog]: This master's current role is: LEADER
W20260620 00:06:57.496709 3385 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 70e5675520ca4ef68f656ab79611adaa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:06:57.496819 3385 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:06:57.496953 3376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:06:57.497191 3376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:06:57.498489 3367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:57.486451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57690 (local address 127.0.98.62:33493)
0620 00:06:57.487484 (+ 1033us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:57.487494 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:57.487530 (+ 36us) server_negotiation.cc:407] Connection header received
0620 00:06:57.488105 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:57.488115 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:57.488186 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:57.488342 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:57.489223 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:57.490435 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:57.491213 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:57.491639 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:57.491760 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:57.494375 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:57.494379 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:57.494380 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:57.494394 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:57.496636 (+ 2242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:57.497060 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:57.497063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:57.497064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:57.497173 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:57.497425 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:57.497426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:57.497426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:57.497746 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:57.498190 (+ 444us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:57.498247 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 00:06:57.498350 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":978,"thread_start_us":146,"threads_started":1}
I20260620 00:06:57.499181 3376 catalog_manager.cc:1347] Generated new cluster ID: 7947ab85b46b46b78e2dcc501709e04b
I20260620 00:06:57.499231 3376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:06:57.499485 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:06:57.512701 3376 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:06:57.513588 3376 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:06:57.522303 3367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:57.517651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57692 (local address 127.0.98.62:33493)
0620 00:06:57.517699 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:57.517701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:57.517831 (+ 130us) server_negotiation.cc:407] Connection header received
0620 00:06:57.517899 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:57.517900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:57.517911 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:57.517988 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:57.518496 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:57.519314 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:57.520498 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:57.520818 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:57.520883 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:57.521753 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:57.521754 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:57.521754 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:06:57.522002 (+ 248us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:57.522023 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:06:57.522030 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:57.522088 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:57.522174 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 00:06:57.522205 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 00:06:57.525947 3376 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70e5675520ca4ef68f656ab79611adaa: Generated new TSK 0
I20260620 00:06:57.526221 3376 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:06:57.528950 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 3296
2026-06-20T00:06:57Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1338 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:06:57 dist-test-slave-grqt krb5kdc[3397](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:06:57 dist-test-slave-grqt krb5kdc[3397](info): set up 2 sockets
Jun 20 00:06:57 dist-test-slave-grqt krb5kdc[3397](info): commencing operation
krb5kdc: starting...
W20260620 00:06:58.593245 392 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:06:58 dist-test-slave-grqt krb5kdc[3397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:06:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:06:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:06:58.721204 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43385
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43385
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:06:58.803839 3410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:06:58.804049 3410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:06:58.804071 3410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:06:58.805258 3410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:06:58.805305 3410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:06:58.805318 3410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:06:58.805331 3410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:06:58.805343 3410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:06:58.806900 3410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:06:58.807163 3410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:06:58.807410 3410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:06:58.812458 3416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:06:58.812558 3418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:06:58.812603 3410 server_base.cc:1034] running on GCE node
W20260620 00:06:58.812565 3415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:06:58.812984 3410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:06:58.813333 3410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:06:58.814565 3410 hybrid_clock.cc:648] HybridClock initialized: now 1781914018814516 us; error 43 us; skew 500 ppm
Jun 20 00:06:58 dist-test-slave-grqt krb5kdc[3397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:06:58.817318 3410 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:06:58.817890 3410 webserver.cc:460] Webserver started at http://127.0.98.62:39581/ using document root <none> and password file <none>
I20260620 00:06:58.818147 3410 fs_manager.cc:359] Metadata directory not provided
I20260620 00:06:58.818220 3410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:06:58.818372 3410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:06:58.819463 3410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "ec28e25cae0e4d2f872ff38548cd0660"
format_stamp: "Formatted at 2026-06-20 00:06:58 on dist-test-slave-grqt"
server_key: "3c492bc390bb5aabe96b4fe739eb6d7c"
server_key_iv: "fda60b583c6d346ebeafc9e66443187c"
server_key_version: "encryptionkey@0"
I20260620 00:06:58.819832 3410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "ec28e25cae0e4d2f872ff38548cd0660"
format_stamp: "Formatted at 2026-06-20 00:06:58 on dist-test-slave-grqt"
server_key: "3c492bc390bb5aabe96b4fe739eb6d7c"
server_key_iv: "fda60b583c6d346ebeafc9e66443187c"
server_key_version: "encryptionkey@0"
I20260620 00:06:58.821287 3410 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:06:58.822119 3425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:58.822307 3410 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:06:58.822412 3410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "ec28e25cae0e4d2f872ff38548cd0660"
format_stamp: "Formatted at 2026-06-20 00:06:58 on dist-test-slave-grqt"
server_key: "3c492bc390bb5aabe96b4fe739eb6d7c"
server_key_iv: "fda60b583c6d346ebeafc9e66443187c"
server_key_version: "encryptionkey@0"
I20260620 00:06:58.822489 3410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:06:58.831672 3410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:06:58.834229 3410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:06:58.834410 3410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:06:58.838761 3410 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43385
I20260620 00:06:58.838793 3477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43385 every 8 connection(s)
I20260620 00:06:58.839175 3410 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:06:58.840245 3478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:06:58.842620 3478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec28e25cae0e4d2f872ff38548cd0660: Bootstrap starting.
I20260620 00:06:58.843233 3478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec28e25cae0e4d2f872ff38548cd0660: Neither blocks nor log segments found. Creating new log.
I20260620 00:06:58.843564 3478 log.cc:824] T 00000000000000000000000000000000 P ec28e25cae0e4d2f872ff38548cd0660: Log is configured to *not* fsync() on all Append() calls
I20260620 00:06:58.844434 3478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec28e25cae0e4d2f872ff38548cd0660: No bootstrap required, opened a new log
I20260620 00:06:58.846199 3478 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec28e25cae0e4d2f872ff38548cd0660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec28e25cae0e4d2f872ff38548cd0660" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43385 } }
I20260620 00:06:58.846340 3478 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec28e25cae0e4d2f872ff38548cd0660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:06:58.846362 3478 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec28e25cae0e4d2f872ff38548cd0660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec28e25cae0e4d2f872ff38548cd0660, State: Initialized, Role: FOLLOWER
I20260620 00:06:58.846323 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 3410
I20260620 00:06:58.846436 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:06:58.846522 3478 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec28e25cae0e4d2f872ff38548cd0660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec28e25cae0e4d2f872ff38548cd0660" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43385 } }
I20260620 00:06:58.846630 3478 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec28e25cae0e4d2f872ff38548cd0660 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:06:58.846688 3478 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec28e25cae0e4d2f872ff38548cd0660 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:06:58.846685 392 external_mini_cluster.cc:1428] Setting key 166301e9ba917081c34165cd13c14756
I20260620 00:06:58.846737 3478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec28e25cae0e4d2f872ff38548cd0660 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:06:58.847503 3478 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec28e25cae0e4d2f872ff38548cd0660 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec28e25cae0e4d2f872ff38548cd0660" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43385 } }
I20260620 00:06:58.847599 3478 leader_election.cc:302] T 00000000000000000000000000000000 P ec28e25cae0e4d2f872ff38548cd0660 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec28e25cae0e4d2f872ff38548cd0660; no voters:
I20260620 00:06:58.847777 3478 leader_election.cc:288] T 00000000000000000000000000000000 P ec28e25cae0e4d2f872ff38548cd0660 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:06:58.847821 3483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec28e25cae0e4d2f872ff38548cd0660 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:06:58.847972 3483 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec28e25cae0e4d2f872ff38548cd0660 [term 1 LEADER]: Becoming Leader. State: Replica: ec28e25cae0e4d2f872ff38548cd0660, State: Running, Role: LEADER
I20260620 00:06:58.848059 3478 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec28e25cae0e4d2f872ff38548cd0660 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:06:58.848099 3483 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec28e25cae0e4d2f872ff38548cd0660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec28e25cae0e4d2f872ff38548cd0660" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43385 } }
I20260620 00:06:58.848434 3483 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec28e25cae0e4d2f872ff38548cd0660 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec28e25cae0e4d2f872ff38548cd0660. Latest consensus state: current_term: 1 leader_uuid: "ec28e25cae0e4d2f872ff38548cd0660" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec28e25cae0e4d2f872ff38548cd0660" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43385 } } }
I20260620 00:06:58.848443 3484 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec28e25cae0e4d2f872ff38548cd0660 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec28e25cae0e4d2f872ff38548cd0660" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec28e25cae0e4d2f872ff38548cd0660" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43385 } } }
I20260620 00:06:58.848636 3484 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec28e25cae0e4d2f872ff38548cd0660 [sys.catalog]: This master's current role is: LEADER
I20260620 00:06:58.848839 3483 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec28e25cae0e4d2f872ff38548cd0660 [sys.catalog]: This master's current role is: LEADER
W20260620 00:06:58.850050 3499 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ec28e25cae0e4d2f872ff38548cd0660: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:06:58.850107 3499 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:06:58.850163 3491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:06:58.850337 3491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:06:58.851867 3491 catalog_manager.cc:1347] Generated new cluster ID: f20d8fc069ef4e85aa99fa6dc2f7b82f
I20260620 00:06:58.851926 3491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 00:06:58 dist-test-slave-grqt krb5kdc[3397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:06:58.858089 3482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:58.847222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50938 (local address 127.0.98.62:43385)
0620 00:06:58.847775 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:58.847788 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:58.847840 (+ 52us) server_negotiation.cc:407] Connection header received
0620 00:06:58.847969 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:58.847976 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:58.848030 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:58.848991 (+ 961us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:58.849129 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:58.850603 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:58.851407 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:58.852025 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:58.852154 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:58.854597 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:58.854602 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:58.854602 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:58.854615 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:58.856351 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:58.856775 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:58.856778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:58.856779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:58.856891 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:58.857112 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:58.857113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:58.857113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:58.857458 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:58.857651 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:58.857723 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 00:06:58.857872 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":469,"threads_started":1}
I20260620 00:06:58.859338 3491 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:06:58.859843 3491 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:06:58.866763 3491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec28e25cae0e4d2f872ff38548cd0660: Generated new TSK 0
I20260620 00:06:58.867036 3491 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:06:58.902084 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43385
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:06:58.985186 3504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:06:58.985460 3504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:06:58.985504 3504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:06:58.986706 3504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:06:58.986775 3504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:06:58.986821 3504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:06:58.988276 3504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:06:58.988544 3504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:06:58.988783 3504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:06:58.993878 3512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:06:58.993892 3509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:06:58.993975 3510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:06:58.994141 3504 server_base.cc:1034] running on GCE node
I20260620 00:06:58.994362 3504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:06:58.994563 3504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:06:58.995776 3504 hybrid_clock.cc:648] HybridClock initialized: now 1781914018995763 us; error 28 us; skew 500 ppm
Jun 20 00:06:58 dist-test-slave-grqt krb5kdc[3397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:06:58.998699 3504 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:06:58.999162 3504 webserver.cc:460] Webserver started at http://127.0.98.1:32951/ using document root <none> and password file <none>
I20260620 00:06:58.999411 3504 fs_manager.cc:359] Metadata directory not provided
I20260620 00:06:58.999480 3504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:06:58.999632 3504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:06:59.000696 3504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-0/data/instance:
uuid: "97d45d706a2e46238a7560c719ac92bd"
format_stamp: "Formatted at 2026-06-20 00:06:58 on dist-test-slave-grqt"
server_key: "eb47a0fe95cfe40f422e77a7ab8dd3eb"
server_key_iv: "5a501a1c1ed72741694a88bbea62433a"
server_key_version: "encryptionkey@0"
I20260620 00:06:59.001058 3504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-0/wal/instance:
uuid: "97d45d706a2e46238a7560c719ac92bd"
format_stamp: "Formatted at 2026-06-20 00:06:58 on dist-test-slave-grqt"
server_key: "eb47a0fe95cfe40f422e77a7ab8dd3eb"
server_key_iv: "5a501a1c1ed72741694a88bbea62433a"
server_key_version: "encryptionkey@0"
I20260620 00:06:59.002461 3504 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:06:59.003243 3519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:59.003432 3504 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:06:59.003539 3504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-0/wal
uuid: "97d45d706a2e46238a7560c719ac92bd"
format_stamp: "Formatted at 2026-06-20 00:06:58 on dist-test-slave-grqt"
server_key: "eb47a0fe95cfe40f422e77a7ab8dd3eb"
server_key_iv: "5a501a1c1ed72741694a88bbea62433a"
server_key_version: "encryptionkey@0"
I20260620 00:06:59.003638 3504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:06:59.017270 3504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:06:59.019202 3504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:06:59.019363 3504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:06:59.019593 3504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:06:59.019888 3504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:06:59.019942 3504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:59.019984 3504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:06:59.020031 3504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:59.026021 3504 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39057
I20260620 00:06:59.026094 3632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39057 every 8 connection(s)
I20260620 00:06:59.026489 3504 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:06:59.027670 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 3504
I20260620 00:06:59.027763 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:06:59.027953 392 external_mini_cluster.cc:1428] Setting key c16d8ad4bfe5ce2568045d8d81a7f9c1
Jun 20 00:06:59 dist-test-slave-grqt krb5kdc[3397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 00:06:59.037539 3482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:59.027237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52471 (local address 127.0.98.62:43385)
0620 00:06:59.027313 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:59.027316 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:59.027524 (+ 208us) server_negotiation.cc:407] Connection header received
0620 00:06:59.028114 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:59.028116 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:59.028132 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:59.028200 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:59.029222 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:59.030030 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:59.031647 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:59.031841 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:59.031893 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:59.034417 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:59.034419 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:59.034420 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:59.034428 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:59.035999 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:59.036502 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:59.036503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:59.036504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:59.036565 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:59.036830 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:59.036831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:59.036831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:59.036948 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:59.037028 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:59.037317 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 00:06:59.037441 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 00:06:59.037724 3635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:59.027228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43385 (local address 127.0.98.1:52471)
0620 00:06:59.027402 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:59.027438 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:59.027997 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:59.028278 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:59.028281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:59.028927 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:06:59.029121 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:59.029126 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:59.030186 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:59.030187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:59.031560 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:59.031563 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:59.031927 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:59.031929 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:59.032093 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:59.032618 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:06:59.032629 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:06:59.034299 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:06:59.036106 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:59.036113 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:59.036115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:59.036413 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:59.036634 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:59.036635 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:59.036635 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:59.036753 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:59.037017 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:06:59.037024 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:06:59.037173 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 00:06:59.037297 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 00:06:59.037482 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":107,"threads_started":1}
I20260620 00:06:59.038347 3633 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43385
I20260620 00:06:59.038460 3633 heartbeater.cc:464] Registering TS with master...
I20260620 00:06:59.038640 3633 heartbeater.cc:511] Master 127.0.98.62:43385 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 00:06:59.039211 3442 ts_manager.cc:194] Registered new tserver with Master: 97d45d706a2e46238a7560c719ac92bd (127.0.98.1:39057)
I20260620 00:06:59.039829 3442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52471
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:06:59.056527 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43385
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:06:59.138618 3638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:06:59.138851 3638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:06:59.138876 3638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:06:59.140087 3638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:06:59.140136 3638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:06:59.140163 3638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:06:59.141698 3638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:06:59.141949 3638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:06:59.142195 3638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:06:59.147383 3643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:06:59.147485 3638 server_base.cc:1034] running on GCE node
W20260620 00:06:59.147392 3646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:06:59.147392 3644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:06:59.147827 3638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:06:59.148092 3638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:06:59.149245 3638 hybrid_clock.cc:648] HybridClock initialized: now 1781914019149230 us; error 28 us; skew 500 ppm
Jun 20 00:06:59 dist-test-slave-grqt krb5kdc[3397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914019, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:06:59.151958 3638 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:06:59.152365 3638 webserver.cc:460] Webserver started at http://127.0.98.2:39839/ using document root <none> and password file <none>
I20260620 00:06:59.152555 3638 fs_manager.cc:359] Metadata directory not provided
I20260620 00:06:59.152597 3638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:06:59.152685 3638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:06:59.153770 3638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-1/data/instance:
uuid: "b18e8aa5993c4dffa09ab877524e75ee"
format_stamp: "Formatted at 2026-06-20 00:06:59 on dist-test-slave-grqt"
server_key: "dd7e1937238a6b46169aa61a8792f681"
server_key_iv: "d312a7ef352d6265edfda1271d62cb0a"
server_key_version: "encryptionkey@0"
I20260620 00:06:59.154064 3638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b18e8aa5993c4dffa09ab877524e75ee"
format_stamp: "Formatted at 2026-06-20 00:06:59 on dist-test-slave-grqt"
server_key: "dd7e1937238a6b46169aa61a8792f681"
server_key_iv: "d312a7ef352d6265edfda1271d62cb0a"
server_key_version: "encryptionkey@0"
I20260620 00:06:59.155297 3638 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:06:59.156090 3653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:59.156245 3638 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:06:59.156332 3638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-1/wal
uuid: "b18e8aa5993c4dffa09ab877524e75ee"
format_stamp: "Formatted at 2026-06-20 00:06:59 on dist-test-slave-grqt"
server_key: "dd7e1937238a6b46169aa61a8792f681"
server_key_iv: "d312a7ef352d6265edfda1271d62cb0a"
server_key_version: "encryptionkey@0"
I20260620 00:06:59.156391 3638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:06:59.166119 3638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:06:59.168210 3638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:06:59.168329 3638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:06:59.168526 3638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:06:59.168938 3638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:06:59.168977 3638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:59.169000 3638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:06:59.169013 3638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:59.175240 3638 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45553
I20260620 00:06:59.175312 3766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45553 every 8 connection(s)
I20260620 00:06:59.175658 3638 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:06:59.181661 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 3638
I20260620 00:06:59.181782 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:06:59.181995 392 external_mini_cluster.cc:1428] Setting key f754331d09a0416c3cb08c30adb8dcab
Jun 20 00:06:59 dist-test-slave-grqt krb5kdc[3397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914019, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:06:59.185627 3482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:59.176457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59439 (local address 127.0.98.62:43385)
0620 00:06:59.176502 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:59.176504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:59.176714 (+ 210us) server_negotiation.cc:407] Connection header received
0620 00:06:59.177202 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:59.177203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:59.177214 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:59.177275 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:59.178306 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:59.178839 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:59.179866 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:59.180078 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:59.180158 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:59.182673 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:59.182674 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:59.182674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:59.182682 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:59.184106 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:59.184582 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:59.184584 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:59.184584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:59.184652 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:59.184915 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:59.184916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:59.184916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:59.185038 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:59.185115 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:59.185441 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 00:06:59.185532 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 00:06:59.185861 3769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:59.176430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43385 (local address 127.0.98.2:59439)
0620 00:06:59.176579 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:59.176615 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:59.177093 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:59.177333 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:59.177336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:59.177993 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:06:59.178204 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:59.178209 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:59.178935 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:59.178936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:59.179770 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:59.179773 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:59.180240 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:59.180242 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:59.180366 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:59.180857 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:06:59.180865 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:06:59.182547 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:06:59.184212 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:59.184219 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:59.184222 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:59.184475 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:59.184719 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:59.184720 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:59.184720 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:59.184845 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:59.185106 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:06:59.185112 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:06:59.185262 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 00:06:59.185416 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 00:06:59.185591 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
I20260620 00:06:59.186326 3767 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43385
I20260620 00:06:59.186446 3767 heartbeater.cc:464] Registering TS with master...
I20260620 00:06:59.186681 3767 heartbeater.cc:511] Master 127.0.98.62:43385 requested a full tablet report, sending...
I20260620 00:06:59.187094 3442 ts_manager.cc:194] Registered new tserver with Master: b18e8aa5993c4dffa09ab877524e75ee (127.0.98.2:45553)
I20260620 00:06:59.187500 3442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59439
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:06:59.209074 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43385
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:06:59.292188 3772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:06:59.292443 3772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:06:59.292469 3772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:06:59.293710 3772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:06:59.293758 3772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:06:59.293795 3772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:06:59.295276 3772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:06:59.295544 3772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:06:59.295789 3772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:06:59.301065 3780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:06:59.301065 3777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:06:59.301092 3778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:06:59.301515 3772 server_base.cc:1034] running on GCE node
I20260620 00:06:59.301731 3772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:06:59.301985 3772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:06:59.303184 3772 hybrid_clock.cc:648] HybridClock initialized: now 1781914019303168 us; error 33 us; skew 500 ppm
Jun 20 00:06:59 dist-test-slave-grqt krb5kdc[3397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914019, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:06:59.305948 3772 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:06:59.306476 3772 webserver.cc:460] Webserver started at http://127.0.98.3:43275/ using document root <none> and password file <none>
I20260620 00:06:59.306777 3772 fs_manager.cc:359] Metadata directory not provided
I20260620 00:06:59.306849 3772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:06:59.306988 3772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:06:59.308072 3772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-2/data/instance:
uuid: "2da60c582b444b808fadc33fbbb40dbc"
format_stamp: "Formatted at 2026-06-20 00:06:59 on dist-test-slave-grqt"
server_key: "851c6c5470c43891e84c0e20ee6a6331"
server_key_iv: "73e9c330dc10000ef7d6ba80f90c6d90"
server_key_version: "encryptionkey@0"
I20260620 00:06:59.308452 3772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2da60c582b444b808fadc33fbbb40dbc"
format_stamp: "Formatted at 2026-06-20 00:06:59 on dist-test-slave-grqt"
server_key: "851c6c5470c43891e84c0e20ee6a6331"
server_key_iv: "73e9c330dc10000ef7d6ba80f90c6d90"
server_key_version: "encryptionkey@0"
I20260620 00:06:59.309849 3772 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:06:59.310631 3787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:59.310794 3772 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:06:59.310892 3772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-2/wal
uuid: "2da60c582b444b808fadc33fbbb40dbc"
format_stamp: "Formatted at 2026-06-20 00:06:59 on dist-test-slave-grqt"
server_key: "851c6c5470c43891e84c0e20ee6a6331"
server_key_iv: "73e9c330dc10000ef7d6ba80f90c6d90"
server_key_version: "encryptionkey@0"
I20260620 00:06:59.310974 3772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:06:59.325259 3772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:06:59.327705 3772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:06:59.327872 3772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:06:59.328118 3772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:06:59.328423 3772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:06:59.328476 3772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:59.328516 3772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:06:59.328544 3772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:06:59.334664 3772 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33919
I20260620 00:06:59.334704 3900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33919 every 8 connection(s)
I20260620 00:06:59.335031 3772 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:06:59.344192 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 3772
I20260620 00:06:59.344309 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914001714546-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:06:59.344524 392 external_mini_cluster.cc:1428] Setting key af36467e5aee12bbc266240ac440491b
Jun 20 00:06:59 dist-test-slave-grqt krb5kdc[3397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914019, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:06:59.348393 3482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:59.336248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49661 (local address 127.0.98.62:43385)
0620 00:06:59.336291 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:59.336293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:59.336521 (+ 228us) server_negotiation.cc:407] Connection header received
0620 00:06:59.336988 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:59.336989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:59.336998 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:59.337061 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:59.338087 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:59.338594 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:59.339568 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:59.339769 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:59.339826 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:59.345702 (+ 5876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:59.345703 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:59.345704 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:59.345711 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:59.346927 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:59.347362 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:59.347364 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:59.347364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:59.347427 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:59.347709 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:59.347710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:59.347710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:59.347835 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:59.347914 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:59.348206 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 00:06:59.348290 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 00:06:59.348598 3903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:59.336221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43385 (local address 127.0.98.3:49661)
0620 00:06:59.336380 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 00:06:59.336424 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 00:06:59.336905 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:06:59.337117 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:06:59.337121 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:06:59.337791 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:06:59.337992 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:59.337998 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:59.338676 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:59.338676 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:59.339483 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:06:59.339484 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:06:59.339842 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:06:59.339843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:06:59.339937 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:59.343589 (+ 3652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:06:59.343603 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:06:59.345586 (+ 1983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:06:59.347014 (+ 1428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:59.347021 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:59.347024 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:59.347276 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:59.347493 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:06:59.347493 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:06:59.347494 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:06:59.347615 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:06:59.347904 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:06:59.347910 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:06:59.348062 (+ 152us) client_negotiation.cc:769] Sending connection context
0620 00:06:59.348189 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 00:06:59.348356 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
I20260620 00:06:59.349135 3901 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43385
I20260620 00:06:59.349218 3901 heartbeater.cc:464] Registering TS with master...
I20260620 00:06:59.349462 3901 heartbeater.cc:511] Master 127.0.98.62:43385 requested a full tablet report, sending...
I20260620 00:06:59.349926 3442 ts_manager.cc:194] Registered new tserver with Master: 2da60c582b444b808fadc33fbbb40dbc (127.0.98.3:33919)
I20260620 00:06:59.350271 3442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49661
I20260620 00:06:59.356556 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 00:06:59 dist-test-slave-grqt krb5kdc[3397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914019, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 00:06:59 dist-test-slave-grqt krb5kdc[3397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914019, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:06:59.379778 3482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:06:59.372429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50948 (local address 127.0.98.62:43385)
0620 00:06:59.372495 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 00:06:59.372497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:06:59.372607 (+ 110us) server_negotiation.cc:407] Connection header received
0620 00:06:59.372667 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:06:59.372667 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:06:59.372676 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:06:59.372727 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:06:59.373331 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:59.373894 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:59.374657 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:06:59.374849 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:06:59.374907 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:06:59.377250 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:06:59.377253 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:06:59.377254 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:06:59.377265 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:06:59.378573 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:59.378948 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:59.378949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:59.378950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:59.379010 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:06:59.379216 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:06:59.379217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:06:59.379218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:06:59.379373 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:06:59.379483 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 00:06:59.379590 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 00:06:59.379673 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260620 00:06:59.380044 3442 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50948
I20260620 00:06:59.380605 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 3504
I20260620 00:06:59.385063 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 3638
I20260620 00:06:59.389798 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 3772
I20260620 00:06:59.394179 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 3410
2026-06-20T00:06:59Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1866 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:06:59 dist-test-slave-grqt krb5kdc[3913](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:06:59 dist-test-slave-grqt krb5kdc[3913](info): set up 2 sockets
Jun 20 00:06:59 dist-test-slave-grqt krb5kdc[3913](info): commencing operation
krb5kdc: starting...
W20260620 00:07:00.453578 392 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:07:00 dist-test-slave-grqt krb5kdc[3913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:07:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:07:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:07:00.582093 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33827
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33827
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:07:00.674749 3926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:00.675020 3926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:00.675063 3926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:00.676395 3926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:07:00.676473 3926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:00.676498 3926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:00.676519 3926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:07:00.676551 3926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:07:00.678201 3926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:00.678525 3926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:00.678776 3926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:00.684034 3932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:00.684034 3931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:00.684350 3934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:00.684621 3926 server_base.cc:1034] running on GCE node
I20260620 00:07:00.685017 3926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:00.685403 3926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:00.686618 3926 hybrid_clock.cc:648] HybridClock initialized: now 1781914020686592 us; error 89 us; skew 500 ppm
Jun 20 00:07:00 dist-test-slave-grqt krb5kdc[3913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914020, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:00.689254 3926 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:07:00.689781 3926 webserver.cc:460] Webserver started at http://127.0.98.62:34523/ using document root <none> and password file <none>
I20260620 00:07:00.690025 3926 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:00.690079 3926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:00.690228 3926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:00.691409 3926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "779082e894af412da002864b837c3774"
format_stamp: "Formatted at 2026-06-20 00:07:00 on dist-test-slave-grqt"
server_key: "cc6d02256de28f64517c524383e20b79"
server_key_iv: "a16b0a07762bf0eb75196c9695f016c6"
server_key_version: "encryptionkey@0"
I20260620 00:07:00.691785 3926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "779082e894af412da002864b837c3774"
format_stamp: "Formatted at 2026-06-20 00:07:00 on dist-test-slave-grqt"
server_key: "cc6d02256de28f64517c524383e20b79"
server_key_iv: "a16b0a07762bf0eb75196c9695f016c6"
server_key_version: "encryptionkey@0"
I20260620 00:07:00.693256 3926 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:07:00.694243 3941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:00.694501 3926 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:00.694628 3926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "779082e894af412da002864b837c3774"
format_stamp: "Formatted at 2026-06-20 00:07:00 on dist-test-slave-grqt"
server_key: "cc6d02256de28f64517c524383e20b79"
server_key_iv: "a16b0a07762bf0eb75196c9695f016c6"
server_key_version: "encryptionkey@0"
I20260620 00:07:00.694727 3926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:00.710379 3926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:00.712792 3926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:00.712991 3926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:00.717763 3926 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33827
I20260620 00:07:00.717806 3993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33827 every 8 connection(s)
I20260620 00:07:00.718233 3926 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:07:00.719158 3994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:00.721554 3994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 779082e894af412da002864b837c3774: Bootstrap starting.
I20260620 00:07:00.722185 3994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 779082e894af412da002864b837c3774: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:00.722501 3994 log.cc:824] T 00000000000000000000000000000000 P 779082e894af412da002864b837c3774: Log is configured to *not* fsync() on all Append() calls
I20260620 00:07:00.723310 3994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 779082e894af412da002864b837c3774: No bootstrap required, opened a new log
I20260620 00:07:00.724994 3994 raft_consensus.cc:348] T 00000000000000000000000000000000 P 779082e894af412da002864b837c3774 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "779082e894af412da002864b837c3774" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33827 } }
I20260620 00:07:00.725165 3994 raft_consensus.cc:374] T 00000000000000000000000000000000 P 779082e894af412da002864b837c3774 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:00.725190 3994 raft_consensus.cc:729] T 00000000000000000000000000000000 P 779082e894af412da002864b837c3774 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 779082e894af412da002864b837c3774, State: Initialized, Role: FOLLOWER
I20260620 00:07:00.725355 3994 consensus_queue.cc:260] T 00000000000000000000000000000000 P 779082e894af412da002864b837c3774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "779082e894af412da002864b837c3774" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33827 } }
I20260620 00:07:00.725490 3994 raft_consensus.cc:388] T 00000000000000000000000000000000 P 779082e894af412da002864b837c3774 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:07:00.725535 3994 raft_consensus.cc:482] T 00000000000000000000000000000000 P 779082e894af412da002864b837c3774 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:07:00.725620 3994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 779082e894af412da002864b837c3774 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:00.726236 3994 raft_consensus.cc:504] T 00000000000000000000000000000000 P 779082e894af412da002864b837c3774 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "779082e894af412da002864b837c3774" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33827 } }
I20260620 00:07:00.726363 3994 leader_election.cc:302] T 00000000000000000000000000000000 P 779082e894af412da002864b837c3774 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 779082e894af412da002864b837c3774; no voters:
I20260620 00:07:00.726568 3994 leader_election.cc:288] T 00000000000000000000000000000000 P 779082e894af412da002864b837c3774 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:07:00.726635 3997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 779082e894af412da002864b837c3774 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:07:00.726845 3994 sys_catalog.cc:564] T 00000000000000000000000000000000 P 779082e894af412da002864b837c3774 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:07:00.727048 3997 raft_consensus.cc:686] T 00000000000000000000000000000000 P 779082e894af412da002864b837c3774 [term 1 LEADER]: Becoming Leader. State: Replica: 779082e894af412da002864b837c3774, State: Running, Role: LEADER
I20260620 00:07:00.727149 3997 consensus_queue.cc:237] T 00000000000000000000000000000000 P 779082e894af412da002864b837c3774 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "779082e894af412da002864b837c3774" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33827 } }
I20260620 00:07:00.727626 3999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 779082e894af412da002864b837c3774 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 779082e894af412da002864b837c3774. Latest consensus state: current_term: 1 leader_uuid: "779082e894af412da002864b837c3774" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "779082e894af412da002864b837c3774" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33827 } } }
I20260620 00:07:00.727744 3999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 779082e894af412da002864b837c3774 [sys.catalog]: This master's current role is: LEADER
I20260620 00:07:00.727644 3998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 779082e894af412da002864b837c3774 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "779082e894af412da002864b837c3774" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "779082e894af412da002864b837c3774" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33827 } } }
I20260620 00:07:00.727792 3998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 779082e894af412da002864b837c3774 [sys.catalog]: This master's current role is: LEADER
I20260620 00:07:00.727957 4006 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:07:00.728066 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 3926
I20260620 00:07:00.728166 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:07:00.728369 392 external_mini_cluster.cc:1428] Setting key e647280f47c8a54e7b567869a9c82153
I20260620 00:07:00.728497 4006 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:07:00.730317 4006 catalog_manager.cc:1347] Generated new cluster ID: 62ba1a080f3340a6a5e1f5236c4ab9a3
I20260620 00:07:00.730377 4006 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 00:07:00 dist-test-slave-grqt krb5kdc[3913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:07:00.743184 4016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:00.729192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34364 (local address 127.0.98.62:33827)
0620 00:07:00.729426 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:00.729436 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:00.729466 (+ 30us) server_negotiation.cc:407] Connection header received
0620 00:07:00.729586 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:00.729593 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:00.729654 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:00.729809 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:00.730778 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:00.732697 (+ 1919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:00.733923 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:00.734616 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:00.735078 (+ 462us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:00.739009 (+ 3931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:00.739015 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:00.739015 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:00.739032 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:00.741311 (+ 2279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:00.741794 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:00.741798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:00.741798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:00.741929 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:00.742139 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:00.742140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:00.742141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:00.742517 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:00.742687 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:00.742766 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 00:07:00.742920 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":74,"threads_started":1}
I20260620 00:07:00.752188 4006 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:07:00.752774 4006 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:07:00.767689 4006 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 779082e894af412da002864b837c3774: Generated new TSK 0
I20260620 00:07:00.767944 4006 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:07:00.813899 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33827
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:07:00.926208 4020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:00.926486 4020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:00.926517 4020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:00.928138 4020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:00.928205 4020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:00.928238 4020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:07:00.931167 4020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:00.931959 4020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:00.932252 4020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:00.938949 4025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:00.938934 4028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:00.938956 4026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:00.939960 4020 server_base.cc:1034] running on GCE node
I20260620 00:07:00.940399 4020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:00.940719 4020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:00.941952 4020 hybrid_clock.cc:648] HybridClock initialized: now 1781914020941918 us; error 87 us; skew 500 ppm
Jun 20 00:07:00 dist-test-slave-grqt krb5kdc[3913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914020, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:00.945149 4020 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:07:00.945770 4020 webserver.cc:460] Webserver started at http://127.0.98.1:35399/ using document root <none> and password file <none>
I20260620 00:07:00.946053 4020 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:00.946111 4020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:00.946218 4020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:00.947399 4020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-0/data/instance:
uuid: "1b1552d51937477d8192e8e8d929798c"
format_stamp: "Formatted at 2026-06-20 00:07:00 on dist-test-slave-grqt"
server_key: "0e948978f29248d9bdd3fdc629bad378"
server_key_iv: "3f198797b30afad200e67934f4ef7e28"
server_key_version: "encryptionkey@0"
I20260620 00:07:00.947829 4020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1b1552d51937477d8192e8e8d929798c"
format_stamp: "Formatted at 2026-06-20 00:07:00 on dist-test-slave-grqt"
server_key: "0e948978f29248d9bdd3fdc629bad378"
server_key_iv: "3f198797b30afad200e67934f4ef7e28"
server_key_version: "encryptionkey@0"
I20260620 00:07:00.949647 4020 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 00:07:00.950922 4035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:00.951438 4020 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260620 00:07:00.951808 4020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-0/wal
uuid: "1b1552d51937477d8192e8e8d929798c"
format_stamp: "Formatted at 2026-06-20 00:07:00 on dist-test-slave-grqt"
server_key: "0e948978f29248d9bdd3fdc629bad378"
server_key_iv: "3f198797b30afad200e67934f4ef7e28"
server_key_version: "encryptionkey@0"
I20260620 00:07:00.952092 4020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:00.964066 4020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:00.967294 4020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:00.967494 4020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:00.967809 4020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:07:00.968220 4020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:07:00.968284 4020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:00.968333 4020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:07:00.968379 4020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:00.981237 4020 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33167
I20260620 00:07:00.981287 4148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33167 every 8 connection(s)
I20260620 00:07:00.981694 4020 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:07:00.982429 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 4020
I20260620 00:07:00.982673 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:07:00.982941 392 external_mini_cluster.cc:1428] Setting key 24bea352d8b862f397f9d7ec0390f952
Jun 20 00:07:00 dist-test-slave-grqt krb5kdc[3913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914020, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 00:07:00.998905 4016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:00.982832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57121 (local address 127.0.98.62:33827)
0620 00:07:00.982894 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:00.982897 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:00.983201 (+ 304us) server_negotiation.cc:407] Connection header received
0620 00:07:00.984247 (+ 1046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:00.984249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:00.984271 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:00.984329 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:00.985205 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:00.985825 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:00.987541 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:00.987858 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:00.987982 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:00.992780 (+ 4798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:00.992783 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:00.992783 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:00.992792 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:00.996000 (+ 3208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:00.996832 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:00.996833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:00.996834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:00.996905 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:00.997727 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:00.997729 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:00.997729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:00.997885 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:00.997975 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:00.998528 (+ 553us) server_negotiation.cc:299] Negotiation successful
0620 00:07:00.998780 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 00:07:00.999367 4151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:00.982839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33827 (local address 127.0.98.1:57121)
0620 00:07:00.983036 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:00.983083 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:00.984120 (+ 1037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:00.984482 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:00.984497 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:00.984836 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:00.985065 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:00.985070 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:00.985933 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:00.985935 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:00.987331 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:00.987336 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:00.988025 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:00.988027 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:00.988218 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:00.989016 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:00.989030 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:00.992522 (+ 3492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:00.996245 (+ 3723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:00.996274 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:00.996276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:00.996589 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:00.997030 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:00.997032 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:00.997032 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:00.997552 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:00.997978 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:00.997988 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:00.998173 (+ 185us) client_negotiation.cc:769] Sending connection context
0620 00:07:00.998502 (+ 329us) client_negotiation.cc:240] Negotiation successful
0620 00:07:00.998910 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":113,"threads_started":1}
I20260620 00:07:01.000350 4149 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33827
I20260620 00:07:01.000483 4149 heartbeater.cc:464] Registering TS with master...
I20260620 00:07:01.000739 4149 heartbeater.cc:511] Master 127.0.98.62:33827 requested a full tablet report, sending...
I20260620 00:07:01.001492 3958 ts_manager.cc:194] Registered new tserver with Master: 1b1552d51937477d8192e8e8d929798c (127.0.98.1:33167)
I20260620 00:07:01.002399 3958 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57121
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:07:01.032053 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33827
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:07:01.126154 4154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:01.126482 4154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:01.126509 4154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:01.128101 4154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:01.128162 4154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:01.128196 4154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:07:01.130113 4154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:01.130364 4154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:01.130573 4154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:01.136490 4162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:01.136492 4160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:01.136515 4159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:01.136763 4154 server_base.cc:1034] running on GCE node
I20260620 00:07:01.137035 4154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:01.137282 4154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:01.138458 4154 hybrid_clock.cc:648] HybridClock initialized: now 1781914021138436 us; error 43 us; skew 500 ppm
Jun 20 00:07:01 dist-test-slave-grqt krb5kdc[3913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914021, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:01.141248 4154 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:07:01.141805 4154 webserver.cc:460] Webserver started at http://127.0.98.2:44717/ using document root <none> and password file <none>
I20260620 00:07:01.142055 4154 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:01.142104 4154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:01.142203 4154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:01.143316 4154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-1/data/instance:
uuid: "1126efc8afc74a7180b24205fa94396e"
format_stamp: "Formatted at 2026-06-20 00:07:01 on dist-test-slave-grqt"
server_key: "05f0a6a2b17c993e6a391a2c45b75eb1"
server_key_iv: "694ca761b9caa632798d7da02a972929"
server_key_version: "encryptionkey@0"
I20260620 00:07:01.143697 4154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1126efc8afc74a7180b24205fa94396e"
format_stamp: "Formatted at 2026-06-20 00:07:01 on dist-test-slave-grqt"
server_key: "05f0a6a2b17c993e6a391a2c45b75eb1"
server_key_iv: "694ca761b9caa632798d7da02a972929"
server_key_version: "encryptionkey@0"
I20260620 00:07:01.145144 4154 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:07:01.146037 4169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:01.146246 4154 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:01.146350 4154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-1/wal
uuid: "1126efc8afc74a7180b24205fa94396e"
format_stamp: "Formatted at 2026-06-20 00:07:01 on dist-test-slave-grqt"
server_key: "05f0a6a2b17c993e6a391a2c45b75eb1"
server_key_iv: "694ca761b9caa632798d7da02a972929"
server_key_version: "encryptionkey@0"
I20260620 00:07:01.146440 4154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:01.158905 4154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:01.161135 4154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:01.161307 4154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:01.161577 4154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:07:01.161901 4154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:07:01.161957 4154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:01.161996 4154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:07:01.162022 4154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:01.168429 4154 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33273
I20260620 00:07:01.168468 4282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33273 every 8 connection(s)
I20260620 00:07:01.168841 4154 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 00:07:01 dist-test-slave-grqt krb5kdc[3913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914021, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:07:01.178376 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 4154
I20260620 00:07:01.178473 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:07:01.178617 392 external_mini_cluster.cc:1428] Setting key 2fda8c889b56b314401330066f9d749b
I20260620 00:07:01.180428 4016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:01.169794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34965 (local address 127.0.98.62:33827)
0620 00:07:01.169864 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:01.169866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:01.169987 (+ 121us) server_negotiation.cc:407] Connection header received
0620 00:07:01.170993 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:01.170995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:01.171006 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:01.171078 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:01.171793 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:01.172384 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:01.173476 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:01.173707 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:01.173775 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:01.176333 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:01.176337 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:01.176338 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:01.176347 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:01.177964 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:01.179044 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:01.179046 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:01.179046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:01.179123 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:01.179438 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:01.179439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:01.179439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:01.179604 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:01.179698 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:01.180155 (+ 457us) server_negotiation.cc:299] Negotiation successful
0620 00:07:01.180268 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 00:07:01.180840 4285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:01.169681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33827 (local address 127.0.98.2:34965)
0620 00:07:01.169850 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:01.169895 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:01.170843 (+ 948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:01.171158 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:01.171161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:01.171486 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:01.171693 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:01.171697 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:01.172480 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:01.172481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:01.173320 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:01.173322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:01.173822 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:01.173823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:01.173952 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:01.174557 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:01.174567 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:01.176091 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:01.178192 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:01.178201 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:01.178204 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:01.178925 (+ 721us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:01.179222 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:01.179223 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:01.179223 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:01.179354 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:01.179714 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:01.179720 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:01.179960 (+ 240us) client_negotiation.cc:769] Sending connection context
0620 00:07:01.180105 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 00:07:01.180311 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":91,"threads_started":1}
I20260620 00:07:01.181524 4283 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33827
I20260620 00:07:01.181653 4283 heartbeater.cc:464] Registering TS with master...
I20260620 00:07:01.181874 4283 heartbeater.cc:511] Master 127.0.98.62:33827 requested a full tablet report, sending...
I20260620 00:07:01.182387 3958 ts_manager.cc:194] Registered new tserver with Master: 1126efc8afc74a7180b24205fa94396e (127.0.98.2:33273)
I20260620 00:07:01.182755 3958 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34965
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:07:01.218901 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33827
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:07:01.305843 4288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:01.306102 4288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:01.306139 4288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:01.307403 4288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:01.307479 4288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:01.307523 4288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:07:01.309118 4288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:01.309517 4288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:01.309765 4288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:01.314841 4294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:01.314836 4293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:01.314838 4296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:01.315327 4288 server_base.cc:1034] running on GCE node
I20260620 00:07:01.315572 4288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:01.315855 4288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:01.317011 4288 hybrid_clock.cc:648] HybridClock initialized: now 1781914021316996 us; error 38 us; skew 500 ppm
Jun 20 00:07:01 dist-test-slave-grqt krb5kdc[3913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914021, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:01.319814 4288 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:07:01.320308 4288 webserver.cc:460] Webserver started at http://127.0.98.3:38383/ using document root <none> and password file <none>
I20260620 00:07:01.320598 4288 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:01.320650 4288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:01.320788 4288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:01.321935 4288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-2/data/instance:
uuid: "dbe11f1c58184e0c8d5ae123de95d27b"
format_stamp: "Formatted at 2026-06-20 00:07:01 on dist-test-slave-grqt"
server_key: "23c40a66b7aec8979ce77cdf92a3ba36"
server_key_iv: "07da3df37bbfd78489b6ec0ff509cd2f"
server_key_version: "encryptionkey@0"
I20260620 00:07:01.322297 4288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-2/wal/instance:
uuid: "dbe11f1c58184e0c8d5ae123de95d27b"
format_stamp: "Formatted at 2026-06-20 00:07:01 on dist-test-slave-grqt"
server_key: "23c40a66b7aec8979ce77cdf92a3ba36"
server_key_iv: "07da3df37bbfd78489b6ec0ff509cd2f"
server_key_version: "encryptionkey@0"
I20260620 00:07:01.323679 4288 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:07:01.324512 4303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:01.324704 4288 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:01.324811 4288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-2/wal
uuid: "dbe11f1c58184e0c8d5ae123de95d27b"
format_stamp: "Formatted at 2026-06-20 00:07:01 on dist-test-slave-grqt"
server_key: "23c40a66b7aec8979ce77cdf92a3ba36"
server_key_iv: "07da3df37bbfd78489b6ec0ff509cd2f"
server_key_version: "encryptionkey@0"
I20260620 00:07:01.324887 4288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:01.337251 4288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:01.339412 4288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:01.339605 4288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:01.339854 4288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:07:01.340191 4288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:07:01.340250 4288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:01.340296 4288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:07:01.340327 4288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:01.346549 4288 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41601
I20260620 00:07:01.346582 4416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41601 every 8 connection(s)
I20260620 00:07:01.346932 4288 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 00:07:01 dist-test-slave-grqt krb5kdc[3913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914021, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:07:01.354789 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 4288
I20260620 00:07:01.354895 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914001714546-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:07:01.355105 392 external_mini_cluster.cc:1428] Setting key 09ee204c9d84e2bdb6cd56f5b889901c
I20260620 00:07:01.357451 4016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:01.347716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58819 (local address 127.0.98.62:33827)
0620 00:07:01.347775 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:01.347781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:01.347966 (+ 185us) server_negotiation.cc:407] Connection header received
0620 00:07:01.348949 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:01.348951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:01.348961 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:01.349028 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:01.349833 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:01.350376 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:01.351517 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:01.351738 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:01.351806 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:01.354284 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:01.354286 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:01.354286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:01.354294 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:01.356113 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:01.356483 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:01.356484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:01.356484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:01.356537 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:01.356800 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:01.356800 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:01.356801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:01.356911 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:01.356966 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:01.357259 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 00:07:01.357323 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 00:07:01.357626 4419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:01.347658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33827 (local address 127.0.98.3:58819)
0620 00:07:01.347815 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:01.347858 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:01.348830 (+ 972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:01.349130 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:01.349133 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:01.349516 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:01.349727 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:01.349732 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:01.350489 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:01.350490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:01.351418 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:01.351420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:01.351848 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:01.351849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:01.351981 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:01.352559 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:01.352568 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:01.354172 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:01.356177 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:01.356183 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:01.356185 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:01.356418 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:01.356620 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:01.356621 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:01.356622 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:01.356744 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:01.356976 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:01.356981 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:01.357114 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 00:07:01.357212 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 00:07:01.357343 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260620 00:07:01.358100 4417 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33827
I20260620 00:07:01.358189 4417 heartbeater.cc:464] Registering TS with master...
I20260620 00:07:01.358342 4417 heartbeater.cc:511] Master 127.0.98.62:33827 requested a full tablet report, sending...
I20260620 00:07:01.358637 3958 ts_manager.cc:194] Registered new tserver with Master: dbe11f1c58184e0c8d5ae123de95d27b (127.0.98.3:41601)
I20260620 00:07:01.359009 3958 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58819
I20260620 00:07:01.367395 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 00:07:01 dist-test-slave-grqt krb5kdc[3913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 00:07:01 dist-test-slave-grqt krb5kdc[3913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 00:07:01.391117 4424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:01.383043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46708 (local address 127.0.98.1:33167)
0620 00:07:01.383256 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:01.383261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:01.383274 (+ 13us) server_negotiation.cc:407] Connection header received
0620 00:07:01.383302 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:01.383304 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:01.383319 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:01.383406 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:01.384089 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:01.384850 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:01.385677 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:01.386030 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:01.386097 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:01.388310 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:01.388319 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:01.388319 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:01.388326 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:01.390017 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:01.390335 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:01.390337 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:01.390338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:01.390404 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:01.390620 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:01.390621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:01.390621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:01.390776 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:01.390892 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:01.390972 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 00:07:01.391037 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":122,"threads_started":1}
Jun 20 00:07:01 dist-test-slave-grqt krb5kdc[3913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:07:01.400300 4016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:01.392511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34376 (local address 127.0.98.62:33827)
0620 00:07:01.392588 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:01.392590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:01.392648 (+ 58us) server_negotiation.cc:407] Connection header received
0620 00:07:01.392685 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:01.392686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:01.392696 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:01.392745 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:01.393258 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:01.393820 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:01.394558 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:01.394779 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:01.394852 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:01.397474 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:01.397476 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:01.397476 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:01.397485 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:01.399084 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:01.399511 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:01.399511 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:01.399512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:01.399566 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:01.399737 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:01.399737 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:01.399738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:01.399886 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:01.399962 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:01.400071 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 00:07:01.400176 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 00:07:01.400521 3958 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34376
I20260620 00:07:01.400876 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 4020
I20260620 00:07:01.405541 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 4154
I20260620 00:07:01.410105 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 4288
I20260620 00:07:01.414512 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 3926
2026-06-20T00:07:01Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2020 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:07:01 dist-test-slave-grqt krb5kdc[4429](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:07:01 dist-test-slave-grqt krb5kdc[4429](info): set up 2 sockets
Jun 20 00:07:01 dist-test-slave-grqt krb5kdc[4429](info): commencing operation
krb5kdc: starting...
W20260620 00:07:02.497555 392 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/krb5kdc/test-user.keytab.
W20260620 00:07:03.015682 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.377s user 0.000s sys 0.002s
Jun 20 00:07:03 dist-test-slave-grqt krb5kdc[4429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:07:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:07:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:07:03.068982 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41659
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:07:03.158540 4442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:03.158792 4442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:03.158818 4442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:03.160123 4442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:07:03.160175 4442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:03.160192 4442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:03.160204 4442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:07:03.160216 4442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:07:03.161840 4442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:03.162130 4442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:03.162384 4442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:03.167677 4450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:03.167630 4448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:03.167640 4447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:03.168051 4442 server_base.cc:1034] running on GCE node
I20260620 00:07:03.168386 4442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:03.168747 4442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:03.169957 4442 hybrid_clock.cc:648] HybridClock initialized: now 1781914023169933 us; error 40 us; skew 500 ppm
Jun 20 00:07:03 dist-test-slave-grqt krb5kdc[4429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:03.172919 4442 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:07:03.173326 4442 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:03.173444 4442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:03.173623 4442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:03.174981 4442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "5e9898e657f74606996b249d369ad0fa"
format_stamp: "Formatted at 2026-06-20 00:07:03 on dist-test-slave-grqt"
server_key: "164c627317d19f67ab4af336969c323f"
server_key_iv: "3b3b66f217e0ff89900ce4586cf43754"
server_key_version: "encryptionkey@0"
I20260620 00:07:03.175401 4442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "5e9898e657f74606996b249d369ad0fa"
format_stamp: "Formatted at 2026-06-20 00:07:03 on dist-test-slave-grqt"
server_key: "164c627317d19f67ab4af336969c323f"
server_key_iv: "3b3b66f217e0ff89900ce4586cf43754"
server_key_version: "encryptionkey@0"
I20260620 00:07:03.176923 4442 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:07:03.177846 4456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:03.178022 4442 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:07:03.178133 4442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "5e9898e657f74606996b249d369ad0fa"
format_stamp: "Formatted at 2026-06-20 00:07:03 on dist-test-slave-grqt"
server_key: "164c627317d19f67ab4af336969c323f"
server_key_iv: "3b3b66f217e0ff89900ce4586cf43754"
server_key_version: "encryptionkey@0"
I20260620 00:07:03.178247 4442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:03.197885 4442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:03.200372 4442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:03.200579 4442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:03.205183 4442 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41659
I20260620 00:07:03.205224 4508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41659 every 8 connection(s)
I20260620 00:07:03.205550 4442 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:07:03.206568 4509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:03.208884 4509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e9898e657f74606996b249d369ad0fa: Bootstrap starting.
I20260620 00:07:03.209553 4509 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e9898e657f74606996b249d369ad0fa: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:03.209863 4509 log.cc:824] T 00000000000000000000000000000000 P 5e9898e657f74606996b249d369ad0fa: Log is configured to *not* fsync() on all Append() calls
I20260620 00:07:03.210654 4509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e9898e657f74606996b249d369ad0fa: No bootstrap required, opened a new log
I20260620 00:07:03.212311 4509 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e9898e657f74606996b249d369ad0fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e9898e657f74606996b249d369ad0fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41659 } }
I20260620 00:07:03.212458 4509 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e9898e657f74606996b249d369ad0fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:03.212481 4509 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e9898e657f74606996b249d369ad0fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e9898e657f74606996b249d369ad0fa, State: Initialized, Role: FOLLOWER
I20260620 00:07:03.212656 4509 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e9898e657f74606996b249d369ad0fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e9898e657f74606996b249d369ad0fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41659 } }
I20260620 00:07:03.212733 4509 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e9898e657f74606996b249d369ad0fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:07:03.212755 4509 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e9898e657f74606996b249d369ad0fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:07:03.212785 4509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e9898e657f74606996b249d369ad0fa [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:03.213761 4509 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e9898e657f74606996b249d369ad0fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e9898e657f74606996b249d369ad0fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41659 } }
I20260620 00:07:03.213877 4509 leader_election.cc:302] T 00000000000000000000000000000000 P 5e9898e657f74606996b249d369ad0fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e9898e657f74606996b249d369ad0fa; no voters:
I20260620 00:07:03.214064 4509 leader_election.cc:288] T 00000000000000000000000000000000 P 5e9898e657f74606996b249d369ad0fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:07:03.214167 4512 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e9898e657f74606996b249d369ad0fa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:07:03.214359 4512 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e9898e657f74606996b249d369ad0fa [term 1 LEADER]: Becoming Leader. State: Replica: 5e9898e657f74606996b249d369ad0fa, State: Running, Role: LEADER
I20260620 00:07:03.214382 4509 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e9898e657f74606996b249d369ad0fa [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:07:03.214761 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 4442
I20260620 00:07:03.214864 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:07:03.215034 392 external_mini_cluster.cc:1428] Setting key 3c6648593dfbb54d8160d91cbcb61815
I20260620 00:07:03.214584 4512 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e9898e657f74606996b249d369ad0fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e9898e657f74606996b249d369ad0fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41659 } }
I20260620 00:07:03.215830 4514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e9898e657f74606996b249d369ad0fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e9898e657f74606996b249d369ad0fa. Latest consensus state: current_term: 1 leader_uuid: "5e9898e657f74606996b249d369ad0fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e9898e657f74606996b249d369ad0fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41659 } } }
I20260620 00:07:03.215924 4514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e9898e657f74606996b249d369ad0fa [sys.catalog]: This master's current role is: LEADER
I20260620 00:07:03.215821 4513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e9898e657f74606996b249d369ad0fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e9898e657f74606996b249d369ad0fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e9898e657f74606996b249d369ad0fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41659 } } }
I20260620 00:07:03.216037 4513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e9898e657f74606996b249d369ad0fa [sys.catalog]: This master's current role is: LEADER
I20260620 00:07:03.216223 4529 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:07:03.216969 4529 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:07:03.218592 4529 catalog_manager.cc:1347] Generated new cluster ID: 24c5dcee641d47d19e531d7e2b8ab3d8
I20260620 00:07:03.218642 4529 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 00:07:03 dist-test-slave-grqt krb5kdc[4429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:07:03.226294 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:03.215568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34392 (local address 127.0.98.62:41659)
0620 00:07:03.215870 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:03.215880 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:03.215941 (+ 61us) server_negotiation.cc:407] Connection header received
0620 00:07:03.216080 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:03.216088 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:03.216173 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:03.216326 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:03.217233 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:03.218562 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:03.219556 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:03.220036 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:03.220189 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:03.222813 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:03.222823 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:03.222824 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:03.222835 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:03.224612 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:03.225023 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:03.225027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:03.225027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:03.225137 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:03.225355 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:03.225356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:03.225356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:03.225657 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:03.225840 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:03.225910 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 00:07:03.226073 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":61,"threads_started":1}
I20260620 00:07:03.233456 4529 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:07:03.234103 4529 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:07:03.238608 4529 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e9898e657f74606996b249d369ad0fa: Generated new TSK 0
I20260620 00:07:03.238833 4529 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:07:03.269519 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41659
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:07:03.360852 4535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:03.361078 4535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:03.361105 4535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:03.362494 4535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:03.362551 4535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:03.362582 4535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:07:03.364317 4535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:03.364603 4535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:03.364851 4535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:03.370417 4541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:03.370383 4543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:03.370442 4540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:03.370505 4535 server_base.cc:1034] running on GCE node
I20260620 00:07:03.370895 4535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:03.371218 4535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:03.372458 4535 hybrid_clock.cc:648] HybridClock initialized: now 1781914023372476 us; error 89 us; skew 500 ppm
Jun 20 00:07:03 dist-test-slave-grqt krb5kdc[4429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:03.375725 4535 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:07:03.376119 4535 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:03.376168 4535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:03.376318 4535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:03.377532 4535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-0/data/instance:
uuid: "634a9a1a09984861bf370090b238b3eb"
format_stamp: "Formatted at 2026-06-20 00:07:03 on dist-test-slave-grqt"
server_key: "88299bc63b3c01382dab36d69eed1b20"
server_key_iv: "5147711cbb548df88ae87fa731105618"
server_key_version: "encryptionkey@0"
I20260620 00:07:03.377912 4535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-0/wal/instance:
uuid: "634a9a1a09984861bf370090b238b3eb"
format_stamp: "Formatted at 2026-06-20 00:07:03 on dist-test-slave-grqt"
server_key: "88299bc63b3c01382dab36d69eed1b20"
server_key_iv: "5147711cbb548df88ae87fa731105618"
server_key_version: "encryptionkey@0"
I20260620 00:07:03.379288 4535 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 00:07:03.380133 4549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:03.380436 4535 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:07:03.380513 4535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-0/wal
uuid: "634a9a1a09984861bf370090b238b3eb"
format_stamp: "Formatted at 2026-06-20 00:07:03 on dist-test-slave-grqt"
server_key: "88299bc63b3c01382dab36d69eed1b20"
server_key_iv: "5147711cbb548df88ae87fa731105618"
server_key_version: "encryptionkey@0"
I20260620 00:07:03.380606 4535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:03.410882 4535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:03.413304 4535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:03.413504 4535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:03.413769 4535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:07:03.414100 4535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:07:03.414160 4535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:03.414209 4535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:07:03.414256 4535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:03.420037 4535 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38213
I20260620 00:07:03.420140 4662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38213 every 8 connection(s)
I20260620 00:07:03.420425 4535 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:07:03.426281 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 4535
I20260620 00:07:03.426417 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:07:03.426669 392 external_mini_cluster.cc:1428] Setting key a203b1ec11162b1207811cfcb4c7310a
Jun 20 00:07:03 dist-test-slave-grqt krb5kdc[4429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:07:03.431023 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:03.421211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51645 (local address 127.0.98.62:41659)
0620 00:07:03.421314 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:03.421316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:03.421502 (+ 186us) server_negotiation.cc:407] Connection header received
0620 00:07:03.422016 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:03.422018 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:03.422028 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:03.422102 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:03.423115 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:03.423638 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:03.424684 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:03.424902 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:03.424980 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:03.427688 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:03.427690 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:03.427690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:03.427699 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:03.429155 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:03.429675 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:03.429677 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:03.429677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:03.429744 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:03.430008 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:03.430009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:03.430009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:03.430136 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:03.430273 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:03.430807 (+ 534us) server_negotiation.cc:299] Negotiation successful
0620 00:07:03.430918 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 00:07:03.431257 4665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:03.421140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41659 (local address 127.0.98.1:51645)
0620 00:07:03.421330 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:03.421401 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:03.421917 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:03.422158 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:03.422161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:03.422442 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:03.422960 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:03.422965 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:03.423756 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:03.423757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:03.424569 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:03.424571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:03.424999 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:03.425000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:03.425111 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:03.425735 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:03.425744 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:03.427588 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:03.429268 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:03.429276 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:03.429278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:03.429585 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:03.429811 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:03.429811 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:03.429812 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:03.429935 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:03.430259 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:03.430265 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:03.430583 (+ 318us) client_negotiation.cc:769] Sending connection context
0620 00:07:03.430758 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 00:07:03.430970 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":95,"threads_started":1}
I20260620 00:07:03.431807 4663 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41659
I20260620 00:07:03.431914 4663 heartbeater.cc:464] Registering TS with master...
I20260620 00:07:03.432121 4663 heartbeater.cc:511] Master 127.0.98.62:41659 requested a full tablet report, sending...
I20260620 00:07:03.432788 4473 ts_manager.cc:194] Registered new tserver with Master: 634a9a1a09984861bf370090b238b3eb (127.0.98.1:38213)
I20260620 00:07:03.433804 4473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51645
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:07:03.458663 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41659
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:07:03.555307 4668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:03.555768 4668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:03.555799 4668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:03.557482 4668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:03.557541 4668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:03.557574 4668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:07:03.559387 4668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:03.559691 4668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:03.559955 4668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:03.566994 4673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:03.567124 4674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:03.567029 4676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:03.567335 4668 server_base.cc:1034] running on GCE node
I20260620 00:07:03.567572 4668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:03.567829 4668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:03.568998 4668 hybrid_clock.cc:648] HybridClock initialized: now 1781914023568970 us; error 41 us; skew 500 ppm
Jun 20 00:07:03 dist-test-slave-grqt krb5kdc[4429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:03.571851 4668 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:07:03.572281 4668 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:03.572333 4668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:03.572467 4668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:03.573679 4668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-1/data/instance:
uuid: "84a5fd64f8124a3cac1115a7f6cb12bd"
format_stamp: "Formatted at 2026-06-20 00:07:03 on dist-test-slave-grqt"
server_key: "f902a96398db48517312fe9c7c194bd1"
server_key_iv: "8836e5ea99921e3b640af0a4b446fda0"
server_key_version: "encryptionkey@0"
I20260620 00:07:03.574101 4668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-1/wal/instance:
uuid: "84a5fd64f8124a3cac1115a7f6cb12bd"
format_stamp: "Formatted at 2026-06-20 00:07:03 on dist-test-slave-grqt"
server_key: "f902a96398db48517312fe9c7c194bd1"
server_key_iv: "8836e5ea99921e3b640af0a4b446fda0"
server_key_version: "encryptionkey@0"
I20260620 00:07:03.575595 4668 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:07:03.576520 4682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:03.576794 4668 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:03.576886 4668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-1/wal
uuid: "84a5fd64f8124a3cac1115a7f6cb12bd"
format_stamp: "Formatted at 2026-06-20 00:07:03 on dist-test-slave-grqt"
server_key: "f902a96398db48517312fe9c7c194bd1"
server_key_iv: "8836e5ea99921e3b640af0a4b446fda0"
server_key_version: "encryptionkey@0"
I20260620 00:07:03.576948 4668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:03.594828 4668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:03.597566 4668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:03.597705 4668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:03.597909 4668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:07:03.598287 4668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:07:03.598327 4668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:03.598354 4668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:07:03.598408 4668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:03.604959 4668 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45807
I20260620 00:07:03.605082 4795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45807 every 8 connection(s)
I20260620 00:07:03.605440 4668 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 00:07:03 dist-test-slave-grqt krb5kdc[4429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:07:03.614684 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 4668
I20260620 00:07:03.614823 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:07:03.615103 392 external_mini_cluster.cc:1428] Setting key d3288349b2f1627b5938d4b6563361fb
I20260620 00:07:03.618011 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:03.606370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47209 (local address 127.0.98.62:41659)
0620 00:07:03.606425 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:03.606427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:03.606641 (+ 214us) server_negotiation.cc:407] Connection header received
0620 00:07:03.607152 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:03.607154 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:03.607165 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:03.607233 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:03.608306 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:03.608942 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:03.610626 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:03.610824 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:03.610890 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:03.613507 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:03.613510 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:03.613510 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:03.613519 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:03.615599 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:03.616569 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:03.616572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:03.616572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:03.616653 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:03.617022 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:03.617023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:03.617023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:03.617193 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:03.617272 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:03.617767 (+ 495us) server_negotiation.cc:299] Negotiation successful
0620 00:07:03.617876 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 00:07:03.618099 4798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:03.606219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41659 (local address 127.0.98.2:47209)
0620 00:07:03.606493 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:03.606532 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:03.607027 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:03.607379 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:03.607384 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:03.607657 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:03.608177 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:03.608182 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:03.609079 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:03.609080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:03.610389 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:03.610393 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:03.610965 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:03.610967 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:03.611094 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:03.611735 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:03.611744 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:03.613317 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:03.615735 (+ 2418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:03.615745 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:03.615748 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:03.616465 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:03.616795 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:03.616796 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:03.616797 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:03.616926 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:03.617315 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:03.617321 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:03.617585 (+ 264us) client_negotiation.cc:769] Sending connection context
0620 00:07:03.617715 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 00:07:03.617875 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":120,"threads_started":1}
I20260620 00:07:03.618732 4796 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41659
I20260620 00:07:03.618863 4796 heartbeater.cc:464] Registering TS with master...
I20260620 00:07:03.619045 4796 heartbeater.cc:511] Master 127.0.98.62:41659 requested a full tablet report, sending...
I20260620 00:07:03.619688 4473 ts_manager.cc:194] Registered new tserver with Master: 84a5fd64f8124a3cac1115a7f6cb12bd (127.0.98.2:45807)
I20260620 00:07:03.620313 4473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47209
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:07:03.650306 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41659
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:07:03.742149 4801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:03.742425 4801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:03.742472 4801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:03.743916 4801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:03.744009 4801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:03.744064 4801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:07:03.745957 4801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:03.746281 4801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:03.746548 4801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:03.752381 4806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:03.752293 4809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:03.752355 4807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:03.752806 4801 server_base.cc:1034] running on GCE node
I20260620 00:07:03.753075 4801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:03.753438 4801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:03.754662 4801 hybrid_clock.cc:648] HybridClock initialized: now 1781914023754635 us; error 46 us; skew 500 ppm
Jun 20 00:07:03 dist-test-slave-grqt krb5kdc[4429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:03.757507 4801 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:07:03.757977 4801 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:03.758055 4801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:03.758206 4801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:03.759322 4801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-2/data/instance:
uuid: "2c62721c0d954e72b2c5f23256c02675"
format_stamp: "Formatted at 2026-06-20 00:07:03 on dist-test-slave-grqt"
server_key: "dfe7cc2477e79ddb5e7c1dc6a39956ba"
server_key_iv: "b70770cf8b69810fe554b1f8658eade3"
server_key_version: "encryptionkey@0"
I20260620 00:07:03.759712 4801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2c62721c0d954e72b2c5f23256c02675"
format_stamp: "Formatted at 2026-06-20 00:07:03 on dist-test-slave-grqt"
server_key: "dfe7cc2477e79ddb5e7c1dc6a39956ba"
server_key_iv: "b70770cf8b69810fe554b1f8658eade3"
server_key_version: "encryptionkey@0"
I20260620 00:07:03.761134 4801 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 00:07:03.762080 4815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:03.762262 4801 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:03.762367 4801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-2/wal
uuid: "2c62721c0d954e72b2c5f23256c02675"
format_stamp: "Formatted at 2026-06-20 00:07:03 on dist-test-slave-grqt"
server_key: "dfe7cc2477e79ddb5e7c1dc6a39956ba"
server_key_iv: "b70770cf8b69810fe554b1f8658eade3"
server_key_version: "encryptionkey@0"
I20260620 00:07:03.762452 4801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:03.779282 4801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:03.782147 4801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:03.782364 4801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:03.782625 4801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:07:03.782956 4801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:07:03.783020 4801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:03.783066 4801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:07:03.783100 4801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:03.789587 4801 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33931
I20260620 00:07:03.789654 4928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33931 every 8 connection(s)
I20260620 00:07:03.789947 4801 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:07:03.796348 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 4801
I20260620 00:07:03.796455 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-2/wal/instance
Jun 20 00:07:03 dist-test-slave-grqt krb5kdc[4429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:07:03.796648 392 external_mini_cluster.cc:1428] Setting key f5cde60e5dcdb7f1745637ec89b37c90
I20260620 00:07:03.800393 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:03.790760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35889 (local address 127.0.98.62:41659)
0620 00:07:03.790876 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:03.790878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:03.791065 (+ 187us) server_negotiation.cc:407] Connection header received
0620 00:07:03.791626 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:03.791627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:03.791641 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:03.791699 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:03.792744 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:03.793255 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:03.794376 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:03.794581 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:03.794661 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:03.797289 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:03.797291 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:03.797291 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:03.797300 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:03.798868 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:03.799330 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:03.799331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:03.799332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:03.799393 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:03.799657 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:03.799658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:03.799658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:03.799781 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:03.799866 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:03.800204 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 00:07:03.800300 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 00:07:03.800614 4931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:03.790738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41659 (local address 127.0.98.3:35889)
0620 00:07:03.790933 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:03.790971 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:03.791504 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:03.791779 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:03.791782 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:03.792048 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:03.792625 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:03.792631 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:03.793363 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:03.793410 (+ 47us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:03.794249 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:03.794251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:03.794668 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:03.794669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:03.794799 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:03.795387 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:03.795395 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:03.797184 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:03.798982 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:03.798990 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:03.798992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:03.799235 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:03.799461 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:03.799462 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:03.799463 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:03.799583 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:03.799856 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:03.799862 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:03.800012 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 00:07:03.800175 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 00:07:03.800346 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":90,"threads_started":1}
I20260620 00:07:03.801065 4929 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41659
I20260620 00:07:03.801183 4929 heartbeater.cc:464] Registering TS with master...
I20260620 00:07:03.801344 4929 heartbeater.cc:511] Master 127.0.98.62:41659 requested a full tablet report, sending...
I20260620 00:07:03.801818 4473 ts_manager.cc:194] Registered new tserver with Master: 2c62721c0d954e72b2c5f23256c02675 (127.0.98.3:33931)
I20260620 00:07:03.802214 4473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35889
I20260620 00:07:03.808768 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:07:03.816726 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:03.810173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34404 (local address 127.0.98.62:41659)
0620 00:07:03.810241 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:03.810243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:03.810302 (+ 59us) server_negotiation.cc:407] Connection header received
0620 00:07:03.810346 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:03.810347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:03.810356 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:03.810408 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:03.811006 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:03.811477 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:03.812338 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:03.812516 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:03.812571 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:03.813871 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:03.813873 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:03.813873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:03.813879 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:03.815520 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:03.815953 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:03.815954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:03.815955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:03.816015 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:03.816229 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:03.816230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:03.816230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:03.816388 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:03.816474 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:03.816583 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 00:07:03.816654 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 00:07:03.818640 4473 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:07:03.820359 4473 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:07:03.827555 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:03.823249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38944 (local address 127.0.98.1:38213)
0620 00:07:03.823814 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:03.823819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:03.823829 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:07:03.823870 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:03.823872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:03.823886 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:03.823962 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:07:03.824272 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:03.824964 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:03.826579 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:03.827089 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:03.827195 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:03.827275 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:03.827419 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 00:07:03.827462 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":69,"threads_started":1}
I20260620 00:07:03.827548 4940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:03.823045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38213 (local address 127.0.0.1:38944)
0620 00:07:03.823343 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:03.823389 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:03.823465 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:03.824052 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:03.824053 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:03.824054 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:07:03.824092 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:03.824093 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:03.825078 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:03.825079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:03.826493 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:03.826495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:03.827200 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:03.827201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:03.827285 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:03.827300 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 00:07:03.827401 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 00:07:03.827462 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":46,"thread_start_us":67,"threads_started":1}
I20260620 00:07:03.827730 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:03.823476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37336 (local address 127.0.98.2:45807)
0620 00:07:03.823683 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:03.823688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:03.823698 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:07:03.823741 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:03.823743 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:03.823758 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:03.823845 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:07:03.824085 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:03.825196 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:03.826427 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:03.827186 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:03.827261 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:03.827318 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:03.827595 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 00:07:03.827641 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":96,"threads_started":1}
I20260620 00:07:03.827729 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:03.823137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55578 (local address 127.0.98.3:33931)
0620 00:07:03.823350 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:03.823357 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:03.823549 (+ 192us) server_negotiation.cc:407] Connection header received
0620 00:07:03.823598 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:03.823600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:03.823617 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:03.823689 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:07:03.823974 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:03.825159 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:03.826512 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:03.827199 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:03.827261 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:03.827317 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:03.827606 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 00:07:03.827644 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":103,"threads_started":1}
I20260620 00:07:03.827920 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:03.823045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33931 (local address 127.0.0.1:55578)
0620 00:07:03.823392 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:03.823402 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:03.823465 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:03.823818 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:03.823821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:03.823823 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:07:03.823894 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:03.823896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:03.825262 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:03.825263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:03.826436 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:03.826438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:03.827359 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:03.827359 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:03.827455 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:03.827459 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:07:03.827503 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 00:07:03.827549 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":33,"thread_start_us":76,"threads_started":1}
I20260620 00:07:03.828114 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:03.823267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45807 (local address 127.0.0.1:37336)
0620 00:07:03.823494 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:03.823505 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:03.823574 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:03.823936 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:03.823937 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:03.823940 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:07:03.823998 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:03.824000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:03.825300 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:03.825301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:03.826327 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:03.826329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:03.827355 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:03.827356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:03.827444 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:03.827449 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:07:03.827501 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 00:07:03.827549 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":3,"thread_start_us":71,"threads_started":1}
I20260620 00:07:03.828768 4597 tablet_service.cc:1467] Processing CreateTablet for tablet 7da3aa4f06164bfb93dc34483e7bc319 (DEFAULT_TABLE table=test-table [id=2cd712ffa9454cee8c69f97bde118221]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:07:03.828805 4730 tablet_service.cc:1467] Processing CreateTablet for tablet 7da3aa4f06164bfb93dc34483e7bc319 (DEFAULT_TABLE table=test-table [id=2cd712ffa9454cee8c69f97bde118221]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:07:03.828811 4863 tablet_service.cc:1467] Processing CreateTablet for tablet 7da3aa4f06164bfb93dc34483e7bc319 (DEFAULT_TABLE table=test-table [id=2cd712ffa9454cee8c69f97bde118221]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:07:03.829133 4597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7da3aa4f06164bfb93dc34483e7bc319. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:03.829136 4730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7da3aa4f06164bfb93dc34483e7bc319. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:03.829133 4863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7da3aa4f06164bfb93dc34483e7bc319. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:03.831949 4946 tablet_bootstrap.cc:492] T 7da3aa4f06164bfb93dc34483e7bc319 P 634a9a1a09984861bf370090b238b3eb: Bootstrap starting.
I20260620 00:07:03.832475 4947 tablet_bootstrap.cc:492] T 7da3aa4f06164bfb93dc34483e7bc319 P 2c62721c0d954e72b2c5f23256c02675: Bootstrap starting.
I20260620 00:07:03.832600 4948 tablet_bootstrap.cc:492] T 7da3aa4f06164bfb93dc34483e7bc319 P 84a5fd64f8124a3cac1115a7f6cb12bd: Bootstrap starting.
I20260620 00:07:03.832901 4946 tablet_bootstrap.cc:654] T 7da3aa4f06164bfb93dc34483e7bc319 P 634a9a1a09984861bf370090b238b3eb: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:03.833153 4947 tablet_bootstrap.cc:654] T 7da3aa4f06164bfb93dc34483e7bc319 P 2c62721c0d954e72b2c5f23256c02675: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:03.833257 4948 tablet_bootstrap.cc:654] T 7da3aa4f06164bfb93dc34483e7bc319 P 84a5fd64f8124a3cac1115a7f6cb12bd: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:03.833606 4946 log.cc:824] T 7da3aa4f06164bfb93dc34483e7bc319 P 634a9a1a09984861bf370090b238b3eb: Log is configured to *not* fsync() on all Append() calls
I20260620 00:07:03.833847 4947 log.cc:824] T 7da3aa4f06164bfb93dc34483e7bc319 P 2c62721c0d954e72b2c5f23256c02675: Log is configured to *not* fsync() on all Append() calls
I20260620 00:07:03.833873 4948 log.cc:824] T 7da3aa4f06164bfb93dc34483e7bc319 P 84a5fd64f8124a3cac1115a7f6cb12bd: Log is configured to *not* fsync() on all Append() calls
I20260620 00:07:03.834443 4946 tablet_bootstrap.cc:492] T 7da3aa4f06164bfb93dc34483e7bc319 P 634a9a1a09984861bf370090b238b3eb: No bootstrap required, opened a new log
I20260620 00:07:03.834525 4946 ts_tablet_manager.cc:1397] T 7da3aa4f06164bfb93dc34483e7bc319 P 634a9a1a09984861bf370090b238b3eb: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260620 00:07:03.834582 4948 tablet_bootstrap.cc:492] T 7da3aa4f06164bfb93dc34483e7bc319 P 84a5fd64f8124a3cac1115a7f6cb12bd: No bootstrap required, opened a new log
I20260620 00:07:03.834664 4948 ts_tablet_manager.cc:1397] T 7da3aa4f06164bfb93dc34483e7bc319 P 84a5fd64f8124a3cac1115a7f6cb12bd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 00:07:03.834973 4947 tablet_bootstrap.cc:492] T 7da3aa4f06164bfb93dc34483e7bc319 P 2c62721c0d954e72b2c5f23256c02675: No bootstrap required, opened a new log
I20260620 00:07:03.835032 4947 ts_tablet_manager.cc:1397] T 7da3aa4f06164bfb93dc34483e7bc319 P 2c62721c0d954e72b2c5f23256c02675: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 00:07:03.836105 4948 raft_consensus.cc:348] T 7da3aa4f06164bfb93dc34483e7bc319 P 84a5fd64f8124a3cac1115a7f6cb12bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c62721c0d954e72b2c5f23256c02675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33931 } } peers { permanent_uuid: "634a9a1a09984861bf370090b238b3eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38213 } } peers { permanent_uuid: "84a5fd64f8124a3cac1115a7f6cb12bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45807 } }
I20260620 00:07:03.836244 4948 raft_consensus.cc:374] T 7da3aa4f06164bfb93dc34483e7bc319 P 84a5fd64f8124a3cac1115a7f6cb12bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:03.836266 4948 raft_consensus.cc:729] T 7da3aa4f06164bfb93dc34483e7bc319 P 84a5fd64f8124a3cac1115a7f6cb12bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84a5fd64f8124a3cac1115a7f6cb12bd, State: Initialized, Role: FOLLOWER
I20260620 00:07:03.836397 4948 consensus_queue.cc:260] T 7da3aa4f06164bfb93dc34483e7bc319 P 84a5fd64f8124a3cac1115a7f6cb12bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c62721c0d954e72b2c5f23256c02675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33931 } } peers { permanent_uuid: "634a9a1a09984861bf370090b238b3eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38213 } } peers { permanent_uuid: "84a5fd64f8124a3cac1115a7f6cb12bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45807 } }
I20260620 00:07:03.836688 4796 heartbeater.cc:503] Master 127.0.98.62:41659 was elected leader, sending a full tablet report...
I20260620 00:07:03.836709 4948 ts_tablet_manager.cc:1428] T 7da3aa4f06164bfb93dc34483e7bc319 P 84a5fd64f8124a3cac1115a7f6cb12bd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 00:07:03.836658 4946 raft_consensus.cc:348] T 7da3aa4f06164bfb93dc34483e7bc319 P 634a9a1a09984861bf370090b238b3eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c62721c0d954e72b2c5f23256c02675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33931 } } peers { permanent_uuid: "634a9a1a09984861bf370090b238b3eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38213 } } peers { permanent_uuid: "84a5fd64f8124a3cac1115a7f6cb12bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45807 } }
I20260620 00:07:03.836879 4946 raft_consensus.cc:374] T 7da3aa4f06164bfb93dc34483e7bc319 P 634a9a1a09984861bf370090b238b3eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:03.836941 4946 raft_consensus.cc:729] T 7da3aa4f06164bfb93dc34483e7bc319 P 634a9a1a09984861bf370090b238b3eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 634a9a1a09984861bf370090b238b3eb, State: Initialized, Role: FOLLOWER
I20260620 00:07:03.837091 4946 consensus_queue.cc:260] T 7da3aa4f06164bfb93dc34483e7bc319 P 634a9a1a09984861bf370090b238b3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c62721c0d954e72b2c5f23256c02675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33931 } } peers { permanent_uuid: "634a9a1a09984861bf370090b238b3eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38213 } } peers { permanent_uuid: "84a5fd64f8124a3cac1115a7f6cb12bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45807 } }
I20260620 00:07:03.837029 4947 raft_consensus.cc:348] T 7da3aa4f06164bfb93dc34483e7bc319 P 2c62721c0d954e72b2c5f23256c02675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c62721c0d954e72b2c5f23256c02675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33931 } } peers { permanent_uuid: "634a9a1a09984861bf370090b238b3eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38213 } } peers { permanent_uuid: "84a5fd64f8124a3cac1115a7f6cb12bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45807 } }
I20260620 00:07:03.837203 4947 raft_consensus.cc:374] T 7da3aa4f06164bfb93dc34483e7bc319 P 2c62721c0d954e72b2c5f23256c02675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:03.837267 4947 raft_consensus.cc:729] T 7da3aa4f06164bfb93dc34483e7bc319 P 2c62721c0d954e72b2c5f23256c02675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c62721c0d954e72b2c5f23256c02675, State: Initialized, Role: FOLLOWER
I20260620 00:07:03.837419 4946 ts_tablet_manager.cc:1428] T 7da3aa4f06164bfb93dc34483e7bc319 P 634a9a1a09984861bf370090b238b3eb: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 00:07:03.837437 4663 heartbeater.cc:503] Master 127.0.98.62:41659 was elected leader, sending a full tablet report...
I20260620 00:07:03.837548 4947 consensus_queue.cc:260] T 7da3aa4f06164bfb93dc34483e7bc319 P 2c62721c0d954e72b2c5f23256c02675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c62721c0d954e72b2c5f23256c02675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33931 } } peers { permanent_uuid: "634a9a1a09984861bf370090b238b3eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38213 } } peers { permanent_uuid: "84a5fd64f8124a3cac1115a7f6cb12bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45807 } }
I20260620 00:07:03.837818 4947 ts_tablet_manager.cc:1428] T 7da3aa4f06164bfb93dc34483e7bc319 P 2c62721c0d954e72b2c5f23256c02675: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 00:07:03.837941 4929 heartbeater.cc:503] Master 127.0.98.62:41659 was elected leader, sending a full tablet report...
W20260620 00:07:03.856056 4797 tablet.cc:2369] T 7da3aa4f06164bfb93dc34483e7bc319 P 84a5fd64f8124a3cac1115a7f6cb12bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 00:07:03.921247 4664 tablet.cc:2369] T 7da3aa4f06164bfb93dc34483e7bc319 P 634a9a1a09984861bf370090b238b3eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:07:04.034017 4953 raft_consensus.cc:482] T 7da3aa4f06164bfb93dc34483e7bc319 P 634a9a1a09984861bf370090b238b3eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:07:04.034162 4953 raft_consensus.cc:504] T 7da3aa4f06164bfb93dc34483e7bc319 P 634a9a1a09984861bf370090b238b3eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c62721c0d954e72b2c5f23256c02675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33931 } } peers { permanent_uuid: "634a9a1a09984861bf370090b238b3eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38213 } } peers { permanent_uuid: "84a5fd64f8124a3cac1115a7f6cb12bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45807 } }
I20260620 00:07:04.034650 4953 leader_election.cc:288] T 7da3aa4f06164bfb93dc34483e7bc319 P 634a9a1a09984861bf370090b238b3eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c62721c0d954e72b2c5f23256c02675 (127.0.98.3:33931), 84a5fd64f8124a3cac1115a7f6cb12bd (127.0.98.2:45807)
I20260620 00:07:04.037917 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:04.035172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45807 (local address 127.0.98.1:59035)
0620 00:07:04.035313 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:04.035325 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:04.035398 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:04.035544 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:04.035544 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:04.035546 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:07:04.035607 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:04.035609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:04.036263 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:04.036264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:04.037064 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:04.037067 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:04.037625 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:04.037626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:04.037699 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:04.037703 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:07:04.037768 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 00:07:04.037803 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":61,"threads_started":1}
I20260620 00:07:04.037906 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:04.035251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59035 (local address 127.0.98.2:45807)
0620 00:07:04.035323 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:04.035325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:04.035389 (+ 64us) server_negotiation.cc:407] Connection header received
0620 00:07:04.035441 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:04.035442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:04.035455 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:04.035511 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:07:04.035679 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:04.036157 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:04.037157 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:04.037542 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:04.037602 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:04.037613 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:04.037788 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 00:07:04.037836 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 00:07:04.037979 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:04.035045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56849 (local address 127.0.98.3:33931)
0620 00:07:04.035114 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:04.035116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:04.035130 (+ 14us) server_negotiation.cc:407] Connection header received
0620 00:07:04.035190 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:04.035191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:04.035203 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:04.035288 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:07:04.035526 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:04.036068 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:04.037087 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:04.037569 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:04.037660 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:04.037669 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:04.037860 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 00:07:04.037902 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 00:07:04.038033 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:04.034905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33931 (local address 127.0.98.1:56849)
0620 00:07:04.035050 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:04.035062 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:04.035123 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:04.035309 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:04.035310 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:04.035313 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:07:04.035429 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:04.035431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:04.036155 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:04.036156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:04.036986 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:04.036989 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:04.037688 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:04.037689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:04.037772 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:04.037780 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 00:07:04.037837 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 00:07:04.037876 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":1,"thread_start_us":85,"threads_started":1}
I20260620 00:07:04.038561 4883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da3aa4f06164bfb93dc34483e7bc319" candidate_uuid: "634a9a1a09984861bf370090b238b3eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c62721c0d954e72b2c5f23256c02675" is_pre_election: true
I20260620 00:07:04.038561 4750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da3aa4f06164bfb93dc34483e7bc319" candidate_uuid: "634a9a1a09984861bf370090b238b3eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84a5fd64f8124a3cac1115a7f6cb12bd" is_pre_election: true
I20260620 00:07:04.038731 4883 raft_consensus.cc:2454] T 7da3aa4f06164bfb93dc34483e7bc319 P 2c62721c0d954e72b2c5f23256c02675 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 634a9a1a09984861bf370090b238b3eb in term 0.
I20260620 00:07:04.038731 4750 raft_consensus.cc:2454] T 7da3aa4f06164bfb93dc34483e7bc319 P 84a5fd64f8124a3cac1115a7f6cb12bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 634a9a1a09984861bf370090b238b3eb in term 0.
I20260620 00:07:04.038975 4553 leader_election.cc:302] T 7da3aa4f06164bfb93dc34483e7bc319 P 634a9a1a09984861bf370090b238b3eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c62721c0d954e72b2c5f23256c02675, 634a9a1a09984861bf370090b238b3eb; no voters:
I20260620 00:07:04.039115 4953 raft_consensus.cc:2781] T 7da3aa4f06164bfb93dc34483e7bc319 P 634a9a1a09984861bf370090b238b3eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:07:04.039209 4953 raft_consensus.cc:482] T 7da3aa4f06164bfb93dc34483e7bc319 P 634a9a1a09984861bf370090b238b3eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:07:04.039247 4953 raft_consensus.cc:3037] T 7da3aa4f06164bfb93dc34483e7bc319 P 634a9a1a09984861bf370090b238b3eb [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:04.039989 4953 raft_consensus.cc:504] T 7da3aa4f06164bfb93dc34483e7bc319 P 634a9a1a09984861bf370090b238b3eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c62721c0d954e72b2c5f23256c02675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33931 } } peers { permanent_uuid: "634a9a1a09984861bf370090b238b3eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38213 } } peers { permanent_uuid: "84a5fd64f8124a3cac1115a7f6cb12bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45807 } }
I20260620 00:07:04.040162 4953 leader_election.cc:288] T 7da3aa4f06164bfb93dc34483e7bc319 P 634a9a1a09984861bf370090b238b3eb [CANDIDATE]: Term 1 election: Requested vote from peers 2c62721c0d954e72b2c5f23256c02675 (127.0.98.3:33931), 84a5fd64f8124a3cac1115a7f6cb12bd (127.0.98.2:45807)
I20260620 00:07:04.040407 4750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da3aa4f06164bfb93dc34483e7bc319" candidate_uuid: "634a9a1a09984861bf370090b238b3eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84a5fd64f8124a3cac1115a7f6cb12bd"
I20260620 00:07:04.040401 4883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da3aa4f06164bfb93dc34483e7bc319" candidate_uuid: "634a9a1a09984861bf370090b238b3eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c62721c0d954e72b2c5f23256c02675"
I20260620 00:07:04.040511 4883 raft_consensus.cc:3037] T 7da3aa4f06164bfb93dc34483e7bc319 P 2c62721c0d954e72b2c5f23256c02675 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:04.040510 4750 raft_consensus.cc:3037] T 7da3aa4f06164bfb93dc34483e7bc319 P 84a5fd64f8124a3cac1115a7f6cb12bd [term 0 FOLLOWER]: Advancing to term 1
W20260620 00:07:04.040551 4930 tablet.cc:2369] T 7da3aa4f06164bfb93dc34483e7bc319 P 2c62721c0d954e72b2c5f23256c02675: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:07:04.041229 4750 raft_consensus.cc:2454] T 7da3aa4f06164bfb93dc34483e7bc319 P 84a5fd64f8124a3cac1115a7f6cb12bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 634a9a1a09984861bf370090b238b3eb in term 1.
I20260620 00:07:04.041345 4883 raft_consensus.cc:2454] T 7da3aa4f06164bfb93dc34483e7bc319 P 2c62721c0d954e72b2c5f23256c02675 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 634a9a1a09984861bf370090b238b3eb in term 1.
I20260620 00:07:04.041421 4553 leader_election.cc:302] T 7da3aa4f06164bfb93dc34483e7bc319 P 634a9a1a09984861bf370090b238b3eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 634a9a1a09984861bf370090b238b3eb, 84a5fd64f8124a3cac1115a7f6cb12bd; no voters:
I20260620 00:07:04.041502 4953 raft_consensus.cc:2781] T 7da3aa4f06164bfb93dc34483e7bc319 P 634a9a1a09984861bf370090b238b3eb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:07:04.041666 4953 raft_consensus.cc:686] T 7da3aa4f06164bfb93dc34483e7bc319 P 634a9a1a09984861bf370090b238b3eb [term 1 LEADER]: Becoming Leader. State: Replica: 634a9a1a09984861bf370090b238b3eb, State: Running, Role: LEADER
I20260620 00:07:04.041785 4953 consensus_queue.cc:237] T 7da3aa4f06164bfb93dc34483e7bc319 P 634a9a1a09984861bf370090b238b3eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c62721c0d954e72b2c5f23256c02675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33931 } } peers { permanent_uuid: "634a9a1a09984861bf370090b238b3eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38213 } } peers { permanent_uuid: "84a5fd64f8124a3cac1115a7f6cb12bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45807 } }
I20260620 00:07:04.042665 4472 catalog_manager.cc:5495] T 7da3aa4f06164bfb93dc34483e7bc319 P 634a9a1a09984861bf370090b238b3eb reported cstate change: term changed from 0 to 1, leader changed from <none> to 634a9a1a09984861bf370090b238b3eb (127.0.98.1). New cstate: current_term: 1 leader_uuid: "634a9a1a09984861bf370090b238b3eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c62721c0d954e72b2c5f23256c02675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "634a9a1a09984861bf370090b238b3eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84a5fd64f8124a3cac1115a7f6cb12bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45807 } health_report { overall_health: UNKNOWN } } }
I20260620 00:07:04.055938 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:04.052990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38952 (local address 127.0.98.1:38213)
0620 00:07:04.053054 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:04.053055 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:04.053063 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:07:04.053096 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:04.053097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:04.053116 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:04.053207 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:07:04.053442 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:04.053902 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:04.054795 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:04.055000 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:04.055058 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:04.055454 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:07:04.055580 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:07:04.055680 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:04.055770 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 00:07:04.055793 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 00:07:04.058065 4750 raft_consensus.cc:1261] T 7da3aa4f06164bfb93dc34483e7bc319 P 84a5fd64f8124a3cac1115a7f6cb12bd [term 1 FOLLOWER]: Refusing update from remote peer 634a9a1a09984861bf370090b238b3eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:07:04.058060 4883 raft_consensus.cc:1261] T 7da3aa4f06164bfb93dc34483e7bc319 P 2c62721c0d954e72b2c5f23256c02675 [term 1 FOLLOWER]: Refusing update from remote peer 634a9a1a09984861bf370090b238b3eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:07:04.058455 4953 consensus_queue.cc:1035] T 7da3aa4f06164bfb93dc34483e7bc319 P 634a9a1a09984861bf370090b238b3eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c62721c0d954e72b2c5f23256c02675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:07:04.058602 4957 consensus_queue.cc:1035] T 7da3aa4f06164bfb93dc34483e7bc319 P 634a9a1a09984861bf370090b238b3eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "84a5fd64f8124a3cac1115a7f6cb12bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:07:04.060848 4960 mvcc.cc:205] Tried to move back new op lower bound from 7298719842538577920 to 7298719842475798528. Current Snapshot: MvccSnapshot[applied={T|T < 7298719842538577920}]
I20260620 00:07:04.061141 4961 mvcc.cc:205] Tried to move back new op lower bound from 7298719842538577920 to 7298719842475798528. Current Snapshot: MvccSnapshot[applied={T|T < 7298719842538577920}]
I20260620 00:07:04.063730 4472 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:07:04.063876 4472 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:07:04.064687 4472 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5e9898e657f74606996b249d369ad0fa: Sending DeleteTablet for 3 replicas of tablet 7da3aa4f06164bfb93dc34483e7bc319
I20260620 00:07:04.065016 4597 tablet_service.cc:1514] Processing DeleteTablet for tablet 7da3aa4f06164bfb93dc34483e7bc319 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:07:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38944
I20260620 00:07:04.065052 4863 tablet_service.cc:1514] Processing DeleteTablet for tablet 7da3aa4f06164bfb93dc34483e7bc319 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:07:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55578
I20260620 00:07:04.065073 4730 tablet_service.cc:1514] Processing DeleteTablet for tablet 7da3aa4f06164bfb93dc34483e7bc319 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:07:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37336
I20260620 00:07:04.065296 4971 tablet_replica.cc:330] T 7da3aa4f06164bfb93dc34483e7bc319 P 2c62721c0d954e72b2c5f23256c02675: stopping tablet replica
I20260620 00:07:04.065438 4971 raft_consensus.cc:2229] T 7da3aa4f06164bfb93dc34483e7bc319 P 2c62721c0d954e72b2c5f23256c02675 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:07:04.065447 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 4535
I20260620 00:07:04.066521 4971 raft_consensus.cc:2258] T 7da3aa4f06164bfb93dc34483e7bc319 P 2c62721c0d954e72b2c5f23256c02675 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:07:04.067174 4971 ts_tablet_manager.cc:1905] T 7da3aa4f06164bfb93dc34483e7bc319 P 2c62721c0d954e72b2c5f23256c02675: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:07:04.069244 4971 ts_tablet_manager.cc:1918] T 7da3aa4f06164bfb93dc34483e7bc319 P 2c62721c0d954e72b2c5f23256c02675: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:07:04.069341 4971 log.cc:1196] T 7da3aa4f06164bfb93dc34483e7bc319 P 2c62721c0d954e72b2c5f23256c02675: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914001714546-392-0/minicluster-data/ts-2/wal/wals/7da3aa4f06164bfb93dc34483e7bc319
I20260620 00:07:04.069638 4971 ts_tablet_manager.cc:1939] T 7da3aa4f06164bfb93dc34483e7bc319 P 2c62721c0d954e72b2c5f23256c02675: Deleting consensus metadata
I20260620 00:07:04.070106 4460 catalog_manager.cc:4841] TS 2c62721c0d954e72b2c5f23256c02675 (127.0.98.3:33931): tablet 7da3aa4f06164bfb93dc34483e7bc319 (table test-table [id=2cd712ffa9454cee8c69f97bde118221]) successfully deleted
W20260620 00:07:04.071834 4457 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38213 (error 108)
I20260620 00:07:04.071969 4972 tablet_replica.cc:330] T 7da3aa4f06164bfb93dc34483e7bc319 P 84a5fd64f8124a3cac1115a7f6cb12bd: stopping tablet replica
I20260620 00:07:04.072109 4972 raft_consensus.cc:2229] T 7da3aa4f06164bfb93dc34483e7bc319 P 84a5fd64f8124a3cac1115a7f6cb12bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:07:04.072252 4972 raft_consensus.cc:2258] T 7da3aa4f06164bfb93dc34483e7bc319 P 84a5fd64f8124a3cac1115a7f6cb12bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:07:04.072448 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 4668
I20260620 00:07:04.072468 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:04.072325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38213 (local address 127.0.0.1:38966)
0620 00:07:04.072378 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:04.072428 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260620 00:07:04.077780 4457 catalog_manager.cc:4584] TS 634a9a1a09984861bf370090b238b3eb (127.0.98.1:38213): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7da3aa4f06164bfb93dc34483e7bc319: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38213: connect: Connection refused (error 111)
I20260620 00:07:04.077998 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:04.077868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45807 (local address 127.0.0.1:37344)
0620 00:07:04.077928 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:04.077959 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 00:07:04.078060 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 4801
I20260620 00:07:04.082927 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 4442
2026-06-20T00:07:04Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2669 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T00:07:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:07:04Z Disabled control of system clock
I20260620 00:07:04.112691 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36249
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 00:07:04.201076 4977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:04.201327 4977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:04.201351 4977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:04.202656 4977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:07:04.202719 4977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:04.202732 4977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:04.202745 4977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:07:04.202757 4977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:07:04.204284 4977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36249
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:04.204581 4977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:04.204926 4977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:04.210397 4982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:04.210373 4985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:04.210605 4977 server_base.cc:1034] running on GCE node
W20260620 00:07:04.210377 4983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:04.210954 4977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:04.211277 4977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:04.212517 4977 hybrid_clock.cc:648] HybridClock initialized: now 1781914024212492 us; error 40 us; skew 500 ppm
I20260620 00:07:04.214094 4977 webserver.cc:460] Webserver started at http://127.0.98.62:36869/ using document root <none> and password file <none>
I20260620 00:07:04.214401 4977 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:04.214483 4977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:04.214643 4977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:04.215790 4977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "2e515fb2e8b841f7a50151e9e7356ddd"
format_stamp: "Formatted at 2026-06-20 00:07:04 on dist-test-slave-grqt"
server_key: "35fee85c4b238780ae6ac93ba7aeff16"
server_key_iv: "d074cfb8193e5703f606dff7f04ea241"
server_key_version: "encryptionkey@0"
I20260620 00:07:04.216176 4977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "2e515fb2e8b841f7a50151e9e7356ddd"
format_stamp: "Formatted at 2026-06-20 00:07:04 on dist-test-slave-grqt"
server_key: "35fee85c4b238780ae6ac93ba7aeff16"
server_key_iv: "d074cfb8193e5703f606dff7f04ea241"
server_key_version: "encryptionkey@0"
I20260620 00:07:04.217700 4977 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:07:04.218619 4991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:04.218817 4977 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:04.218942 4977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "2e515fb2e8b841f7a50151e9e7356ddd"
format_stamp: "Formatted at 2026-06-20 00:07:04 on dist-test-slave-grqt"
server_key: "35fee85c4b238780ae6ac93ba7aeff16"
server_key_iv: "d074cfb8193e5703f606dff7f04ea241"
server_key_version: "encryptionkey@0"
I20260620 00:07:04.219030 4977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:04.223058 4977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:04.223361 4977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:04.223523 4977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:04.227809 4977 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36249
I20260620 00:07:04.227857 5043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36249 every 8 connection(s)
I20260620 00:07:04.228237 4977 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:07:04.228914 5044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:04.231742 5044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e515fb2e8b841f7a50151e9e7356ddd: Bootstrap starting.
I20260620 00:07:04.232309 5044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e515fb2e8b841f7a50151e9e7356ddd: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:04.232599 5044 log.cc:824] T 00000000000000000000000000000000 P 2e515fb2e8b841f7a50151e9e7356ddd: Log is configured to *not* fsync() on all Append() calls
I20260620 00:07:04.233299 5044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e515fb2e8b841f7a50151e9e7356ddd: No bootstrap required, opened a new log
I20260620 00:07:04.234899 5044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e515fb2e8b841f7a50151e9e7356ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e515fb2e8b841f7a50151e9e7356ddd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36249 } }
I20260620 00:07:04.235049 5044 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e515fb2e8b841f7a50151e9e7356ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:04.235072 5044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e515fb2e8b841f7a50151e9e7356ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e515fb2e8b841f7a50151e9e7356ddd, State: Initialized, Role: FOLLOWER
I20260620 00:07:04.235268 5044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e515fb2e8b841f7a50151e9e7356ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e515fb2e8b841f7a50151e9e7356ddd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36249 } }
I20260620 00:07:04.235344 5044 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e515fb2e8b841f7a50151e9e7356ddd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:07:04.235368 5044 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e515fb2e8b841f7a50151e9e7356ddd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:07:04.235402 5044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e515fb2e8b841f7a50151e9e7356ddd [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:04.235899 5044 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e515fb2e8b841f7a50151e9e7356ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e515fb2e8b841f7a50151e9e7356ddd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36249 } }
I20260620 00:07:04.235983 5044 leader_election.cc:302] T 00000000000000000000000000000000 P 2e515fb2e8b841f7a50151e9e7356ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e515fb2e8b841f7a50151e9e7356ddd; no voters:
I20260620 00:07:04.236107 5044 leader_election.cc:288] T 00000000000000000000000000000000 P 2e515fb2e8b841f7a50151e9e7356ddd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:07:04.236230 5047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e515fb2e8b841f7a50151e9e7356ddd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:07:04.236388 5044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e515fb2e8b841f7a50151e9e7356ddd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:07:04.236424 5047 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e515fb2e8b841f7a50151e9e7356ddd [term 1 LEADER]: Becoming Leader. State: Replica: 2e515fb2e8b841f7a50151e9e7356ddd, State: Running, Role: LEADER
I20260620 00:07:04.236522 5047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e515fb2e8b841f7a50151e9e7356ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e515fb2e8b841f7a50151e9e7356ddd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36249 } }
I20260620 00:07:04.236964 5049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e515fb2e8b841f7a50151e9e7356ddd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e515fb2e8b841f7a50151e9e7356ddd. Latest consensus state: current_term: 1 leader_uuid: "2e515fb2e8b841f7a50151e9e7356ddd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e515fb2e8b841f7a50151e9e7356ddd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36249 } } }
I20260620 00:07:04.237061 5049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e515fb2e8b841f7a50151e9e7356ddd [sys.catalog]: This master's current role is: LEADER
I20260620 00:07:04.237208 5048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e515fb2e8b841f7a50151e9e7356ddd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e515fb2e8b841f7a50151e9e7356ddd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e515fb2e8b841f7a50151e9e7356ddd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36249 } } }
I20260620 00:07:04.237285 5048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e515fb2e8b841f7a50151e9e7356ddd [sys.catalog]: This master's current role is: LEADER
I20260620 00:07:04.237474 5056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:07:04.237581 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 4977
I20260620 00:07:04.237682 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:07:04.237916 392 external_mini_cluster.cc:1428] Setting key 1fd4c2766109adaa8440e3118d84d53c
I20260620 00:07:04.238220 5056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:07:04.239944 5056 catalog_manager.cc:1347] Generated new cluster ID: 68b26456a706461b9ffaddb4ab49e226
I20260620 00:07:04.240000 5056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:07:04.240852 5066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:04.238622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50190 (local address 127.0.98.62:36249)
0620 00:07:04.238848 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:04.238856 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:04.239179 (+ 323us) server_negotiation.cc:407] Connection header received
0620 00:07:04.239314 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:04.239323 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:04.239391 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:04.239557 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:04.240360 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:04.240362 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:04.240363 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:07:04.240366 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:04.240403 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:07:04.240414 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:04.240503 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:04.240587 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 00:07:04.240655 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":104,"threads_started":1}
I20260620 00:07:04.249881 5056 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:07:04.250072 5056 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:07:04.255008 5056 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e515fb2e8b841f7a50151e9e7356ddd: Generated new TSK 0
I20260620 00:07:04.255234 5056 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:07:04.258584 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36249
--builtin_ntp_servers=127.0.98.20:36147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 00:07:04.353340 5068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:04.353672 5068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:04.353711 5068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:04.355036 5068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:04.355111 5068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:04.355167 5068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:07:04.356681 5068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:04.356974 5068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:04.357304 5068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:04.363041 5076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:04.363047 5073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:04.363050 5074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:04.364075 5068 server_base.cc:1034] running on GCE node
I20260620 00:07:04.364388 5068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:04.364681 5068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:04.365907 5068 hybrid_clock.cc:648] HybridClock initialized: now 1781914024365884 us; error 42 us; skew 500 ppm
I20260620 00:07:04.371086 5068 webserver.cc:460] Webserver started at http://127.0.98.1:41379/ using document root <none> and password file <none>
I20260620 00:07:04.371410 5068 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:04.371496 5068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:04.371667 5068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:04.372800 5068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-0/data/instance:
uuid: "695f03e59fdb4c64a83cb81e06d01b6a"
format_stamp: "Formatted at 2026-06-20 00:07:04 on dist-test-slave-grqt"
server_key: "31ada3c15c1e23d23b70c43347f0e21e"
server_key_iv: "c0e63fa12d2bce61328e9ef5af794c2b"
server_key_version: "encryptionkey@0"
I20260620 00:07:04.373198 5068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-0/wal/instance:
uuid: "695f03e59fdb4c64a83cb81e06d01b6a"
format_stamp: "Formatted at 2026-06-20 00:07:04 on dist-test-slave-grqt"
server_key: "31ada3c15c1e23d23b70c43347f0e21e"
server_key_iv: "c0e63fa12d2bce61328e9ef5af794c2b"
server_key_version: "encryptionkey@0"
I20260620 00:07:04.375072 5068 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260620 00:07:04.376020 5082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:04.376364 5068 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:07:04.376487 5068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-0/wal
uuid: "695f03e59fdb4c64a83cb81e06d01b6a"
format_stamp: "Formatted at 2026-06-20 00:07:04 on dist-test-slave-grqt"
server_key: "31ada3c15c1e23d23b70c43347f0e21e"
server_key_iv: "c0e63fa12d2bce61328e9ef5af794c2b"
server_key_version: "encryptionkey@0"
I20260620 00:07:04.376586 5068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:04.380497 5068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:04.380823 5068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:04.380978 5068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:04.381237 5068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:07:04.381590 5068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:07:04.381649 5068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:04.381692 5068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:07:04.381721 5068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:04.389412 5068 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40779
I20260620 00:07:04.389465 5195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40779 every 8 connection(s)
I20260620 00:07:04.389873 5068 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:07:04.394353 5066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:04.391004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41335 (local address 127.0.98.62:36249)
0620 00:07:04.391086 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:04.391089 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:04.391493 (+ 404us) server_negotiation.cc:407] Connection header received
0620 00:07:04.392148 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:04.392149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:04.392156 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:04.392343 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:04.394017 (+ 1674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:04.394018 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:04.394018 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:07:04.394023 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:04.394039 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:07:04.394043 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:04.394107 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:04.394258 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 00:07:04.394295 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 00:07:04.394487 5198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:04.391013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36249 (local address 127.0.98.1:41335)
0620 00:07:04.391275 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:04.391337 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:04.391997 (+ 660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:04.392433 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:04.392438 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:04.392450 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:04.393797 (+ 1347us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:07:04.393798 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:04.393843 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:07:04.393885 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:04.394161 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:04.394167 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:04.394170 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:07:04.394258 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 00:07:04.394315 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":110,"threads_started":1}
I20260620 00:07:04.395262 5196 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36249
I20260620 00:07:04.395280 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 5068
I20260620 00:07:04.395434 5196 heartbeater.cc:464] Registering TS with master...
I20260620 00:07:04.395448 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:07:04.395637 5196 heartbeater.cc:511] Master 127.0.98.62:36249 requested a full tablet report, sending...
I20260620 00:07:04.395678 392 external_mini_cluster.cc:1428] Setting key 1b8789eb763409f8115aee196ddac834
I20260620 00:07:04.396256 5008 ts_manager.cc:194] Registered new tserver with Master: 695f03e59fdb4c64a83cb81e06d01b6a (127.0.98.1:40779)
I20260620 00:07:04.397032 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36249
--builtin_ntp_servers=127.0.98.20:36147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 00:07:04.482141 5199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:04.482429 5199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:04.482473 5199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:04.483772 5199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:04.483840 5199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:04.483887 5199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:07:04.485589 5199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:04.485888 5199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:04.486274 5199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:04.491925 5204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:04.491981 5205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:04.492074 5199 server_base.cc:1034] running on GCE node
W20260620 00:07:04.492170 5207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:04.492560 5199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:04.493041 5199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:04.494397 5199 hybrid_clock.cc:648] HybridClock initialized: now 1781914024494339 us; error 80 us; skew 500 ppm
I20260620 00:07:04.495955 5199 webserver.cc:460] Webserver started at http://127.0.98.2:40871/ using document root <none> and password file <none>
I20260620 00:07:04.496253 5199 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:04.496346 5199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:04.496627 5199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:04.498047 5199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-1/data/instance:
uuid: "8853c2e8c61e42889a609fcd895a6eb6"
format_stamp: "Formatted at 2026-06-20 00:07:04 on dist-test-slave-grqt"
server_key: "f306f3f0bd6e90380ba7a8432ee0d351"
server_key_iv: "04a9138493cee4c22e8c1db83898da9e"
server_key_version: "encryptionkey@0"
I20260620 00:07:04.498628 5199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8853c2e8c61e42889a609fcd895a6eb6"
format_stamp: "Formatted at 2026-06-20 00:07:04 on dist-test-slave-grqt"
server_key: "f306f3f0bd6e90380ba7a8432ee0d351"
server_key_iv: "04a9138493cee4c22e8c1db83898da9e"
server_key_version: "encryptionkey@0"
I20260620 00:07:04.500147 5199 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:07:04.501150 5213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:04.501437 5199 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:04.501551 5199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-1/wal
uuid: "8853c2e8c61e42889a609fcd895a6eb6"
format_stamp: "Formatted at 2026-06-20 00:07:04 on dist-test-slave-grqt"
server_key: "f306f3f0bd6e90380ba7a8432ee0d351"
server_key_iv: "04a9138493cee4c22e8c1db83898da9e"
server_key_version: "encryptionkey@0"
I20260620 00:07:04.501644 5199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:04.505088 5199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:04.505448 5199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:04.505610 5199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:04.505832 5199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:07:04.506151 5199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:07:04.506214 5199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:04.506255 5199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:07:04.506282 5199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:04.512588 5199 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35501
I20260620 00:07:04.512630 5326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35501 every 8 connection(s)
I20260620 00:07:04.512985 5199 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:07:04.516324 5066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:04.513861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51975 (local address 127.0.98.62:36249)
0620 00:07:04.513927 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:04.513929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:04.514129 (+ 200us) server_negotiation.cc:407] Connection header received
0620 00:07:04.514699 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:04.514700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:04.514706 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:04.514758 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:04.516046 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:04.516047 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:04.516048 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:07:04.516053 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:04.516070 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:07:04.516075 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:04.516126 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:04.516229 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 00:07:04.516246 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 00:07:04.516538 5329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:04.513828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36249 (local address 127.0.98.2:51975)
0620 00:07:04.514005 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:04.514040 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:04.514571 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:04.514837 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:04.514841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:04.514849 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:04.515873 (+ 1024us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:07:04.515874 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:04.515905 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:07:04.515937 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:04.516132 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:04.516137 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:04.516141 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:07:04.516242 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 00:07:04.516312 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":74,"threads_started":1}
I20260620 00:07:04.517081 5327 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36249
I20260620 00:07:04.517215 5327 heartbeater.cc:464] Registering TS with master...
I20260620 00:07:04.517455 5327 heartbeater.cc:511] Master 127.0.98.62:36249 requested a full tablet report, sending...
I20260620 00:07:04.518026 5008 ts_manager.cc:194] Registered new tserver with Master: 8853c2e8c61e42889a609fcd895a6eb6 (127.0.98.2:35501)
I20260620 00:07:04.522156 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 5199
I20260620 00:07:04.522332 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:07:04.522526 392 external_mini_cluster.cc:1428] Setting key d92cd9da9744ba12218d826904caf97b
I20260620 00:07:04.523831 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36249
--builtin_ntp_servers=127.0.98.20:36147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 00:07:04.609298 5330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:04.609549 5330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:04.609575 5330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:04.610881 5330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:04.610929 5330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:04.610958 5330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:07:04.612453 5330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:04.612757 5330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:04.613085 5330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:04.619242 5336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:04.619240 5335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:04.619234 5338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:04.619529 5330 server_base.cc:1034] running on GCE node
I20260620 00:07:04.619841 5330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:04.620175 5330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:04.621452 5330 hybrid_clock.cc:648] HybridClock initialized: now 1781914024621415 us; error 58 us; skew 500 ppm
I20260620 00:07:04.623227 5330 webserver.cc:460] Webserver started at http://127.0.98.3:40545/ using document root <none> and password file <none>
I20260620 00:07:04.623521 5330 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:04.623615 5330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:04.623777 5330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:04.624963 5330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-2/data/instance:
uuid: "ba1d36c3a4f647b2a04a8b0f895d6904"
format_stamp: "Formatted at 2026-06-20 00:07:04 on dist-test-slave-grqt"
server_key: "f5f23461dab61a9e4b1d2d865fd70a54"
server_key_iv: "fb133ffbf1bee346dde952fef0e22e87"
server_key_version: "encryptionkey@0"
I20260620 00:07:04.625458 5330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ba1d36c3a4f647b2a04a8b0f895d6904"
format_stamp: "Formatted at 2026-06-20 00:07:04 on dist-test-slave-grqt"
server_key: "f5f23461dab61a9e4b1d2d865fd70a54"
server_key_iv: "fb133ffbf1bee346dde952fef0e22e87"
server_key_version: "encryptionkey@0"
I20260620 00:07:04.626951 5330 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:07:04.627996 5344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:04.628253 5330 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:04.628371 5330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-2/wal
uuid: "ba1d36c3a4f647b2a04a8b0f895d6904"
format_stamp: "Formatted at 2026-06-20 00:07:04 on dist-test-slave-grqt"
server_key: "f5f23461dab61a9e4b1d2d865fd70a54"
server_key_iv: "fb133ffbf1bee346dde952fef0e22e87"
server_key_version: "encryptionkey@0"
I20260620 00:07:04.628473 5330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:04.632237 5330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:04.632550 5330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:04.632716 5330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:04.632942 5330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:07:04.633283 5330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:07:04.633342 5330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:04.633378 5330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:07:04.633424 5330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:04.640519 5330 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45901
I20260620 00:07:04.640560 5457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45901 every 8 connection(s)
I20260620 00:07:04.640924 5330 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:07:04.644047 5066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:04.641775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34875 (local address 127.0.98.62:36249)
0620 00:07:04.641845 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:04.641847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:04.642079 (+ 232us) server_negotiation.cc:407] Connection header received
0620 00:07:04.642567 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:04.642568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:04.642574 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:04.642634 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:04.643814 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:04.643815 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:04.643815 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:07:04.643819 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:04.643833 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:07:04.643838 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:04.643892 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:04.643975 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 00:07:04.643989 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 00:07:04.644272 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:04.641767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36249 (local address 127.0.98.3:34875)
0620 00:07:04.641946 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:04.641984 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:04.642450 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:04.642691 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:04.642694 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:04.642704 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:04.643613 (+ 909us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:07:04.643614 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:04.643672 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:07:04.643704 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:04.643908 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:04.643910 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:04.643913 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:07:04.643968 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 00:07:04.644021 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
I20260620 00:07:04.644819 5458 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36249
I20260620 00:07:04.644903 5458 heartbeater.cc:464] Registering TS with master...
I20260620 00:07:04.645076 5458 heartbeater.cc:511] Master 127.0.98.62:36249 requested a full tablet report, sending...
I20260620 00:07:04.645527 5008 ts_manager.cc:194] Registered new tserver with Master: ba1d36c3a4f647b2a04a8b0f895d6904 (127.0.98.3:45901)
I20260620 00:07:04.649160 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 5330
I20260620 00:07:04.649255 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:07:04.649492 392 external_mini_cluster.cc:1428] Setting key dfd81e4bf09c30b4613707ac75fd207e
I20260620 00:07:04.650622 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:07:04.653153 5066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:04.652073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50192 (local address 127.0.98.62:36249)
0620 00:07:04.652138 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:04.652140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:04.652296 (+ 156us) server_negotiation.cc:407] Connection header received
0620 00:07:04.652348 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:04.652349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:04.652354 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:04.652418 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:04.652965 (+ 547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:04.652966 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:04.652966 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:07:04.652970 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:04.652987 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:07:04.652991 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:04.653045 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:04.653079 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 00:07:04.653093 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 00:07:04.655011 5008 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:07:04.656545 5008 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:07:04.660547 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:04.658778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45901 (local address 127.0.0.1:33266)
0620 00:07:04.658959 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:04.659000 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:04.659054 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:04.659704 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:04.659704 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:04.659707 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:04.660149 (+ 442us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:07:04.660149 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:04.660155 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:07:04.660163 (+ 8us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:04.660392 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:04.660398 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:04.660402 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:07:04.660467 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 00:07:04.660481 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":62,"threads_started":1}
I20260620 00:07:04.660769 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:04.658778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40779 (local address 127.0.0.1:34482)
0620 00:07:04.659188 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:04.659199 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:04.659252 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:04.659438 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:04.659444 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:04.659452 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:04.660000 (+ 548us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:07:04.660000 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:04.660041 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:07:04.660081 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:04.660649 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:04.660650 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:04.660651 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 00:07:04.660686 (+ 35us) client_negotiation.cc:240] Negotiation successful
0620 00:07:04.660702 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":2,"spinlock_wait_cycles":768,"thread_start_us":85,"threads_started":1}
I20260620 00:07:04.660804 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:04.658857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33266 (local address 127.0.98.3:45901)
0620 00:07:04.659138 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:04.659142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:04.659152 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:07:04.659176 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:04.659178 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:04.659195 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:04.659291 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:04.660245 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:04.660252 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:04.660252 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:07:04.660257 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:04.660295 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:07:04.660316 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:04.660392 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:04.660687 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 00:07:04.660705 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":107,"threads_started":1}
I20260620 00:07:04.660871 5466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:04.658974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35501 (local address 127.0.0.1:33170)
0620 00:07:04.659318 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:04.659327 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:04.659359 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:04.659809 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:04.659810 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:04.659812 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:04.660374 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:07:04.660375 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:04.660379 (+ 4us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:07:04.660388 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:04.660597 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:04.660598 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:04.660599 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 00:07:04.660771 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 00:07:04.660783 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":68,"threads_started":1}
I20260620 00:07:04.661015 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:04.659161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33170 (local address 127.0.98.2:35501)
0620 00:07:04.659351 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:04.659356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:04.659368 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:07:04.659417 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:04.659419 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:04.659428 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:04.659497 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:04.660449 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:04.660458 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:04.660459 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:07:04.660464 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:04.660497 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:07:04.660524 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:04.660592 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:04.660936 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 00:07:04.660958 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":83,"threads_started":1}
I20260620 00:07:04.661460 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:04.658952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34482 (local address 127.0.98.1:40779)
0620 00:07:04.659104 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:04.659110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:04.659291 (+ 181us) server_negotiation.cc:407] Connection header received
0620 00:07:04.659320 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:04.659322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:04.659331 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:04.659436 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:04.660514 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:04.660517 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:04.660518 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:07:04.660522 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:04.660551 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:07:04.660565 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:04.661290 (+ 725us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:04.661318 (+ 28us) server_negotiation.cc:299] Negotiation successful
0620 00:07:04.661354 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260620 00:07:04.661635 5392 tablet_service.cc:1467] Processing CreateTablet for tablet c0bef085db6241b1b81052bcf7c7f9fa (DEFAULT_TABLE table=test-table [id=267c619075da45f49afb7233d7b18338]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:07:04.661756 5261 tablet_service.cc:1467] Processing CreateTablet for tablet c0bef085db6241b1b81052bcf7c7f9fa (DEFAULT_TABLE table=test-table [id=267c619075da45f49afb7233d7b18338]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:07:04.662010 5261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0bef085db6241b1b81052bcf7c7f9fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:04.662010 5392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0bef085db6241b1b81052bcf7c7f9fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:04.662088 5130 tablet_service.cc:1467] Processing CreateTablet for tablet c0bef085db6241b1b81052bcf7c7f9fa (DEFAULT_TABLE table=test-table [id=267c619075da45f49afb7233d7b18338]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:07:04.662329 5130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0bef085db6241b1b81052bcf7c7f9fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:04.665069 5472 tablet_bootstrap.cc:492] T c0bef085db6241b1b81052bcf7c7f9fa P 695f03e59fdb4c64a83cb81e06d01b6a: Bootstrap starting.
I20260620 00:07:04.665517 5474 tablet_bootstrap.cc:492] T c0bef085db6241b1b81052bcf7c7f9fa P 8853c2e8c61e42889a609fcd895a6eb6: Bootstrap starting.
I20260620 00:07:04.665864 5472 tablet_bootstrap.cc:654] T c0bef085db6241b1b81052bcf7c7f9fa P 695f03e59fdb4c64a83cb81e06d01b6a: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:04.666198 5474 tablet_bootstrap.cc:654] T c0bef085db6241b1b81052bcf7c7f9fa P 8853c2e8c61e42889a609fcd895a6eb6: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:04.666193 5472 log.cc:824] T c0bef085db6241b1b81052bcf7c7f9fa P 695f03e59fdb4c64a83cb81e06d01b6a: Log is configured to *not* fsync() on all Append() calls
I20260620 00:07:04.666512 5474 log.cc:824] T c0bef085db6241b1b81052bcf7c7f9fa P 8853c2e8c61e42889a609fcd895a6eb6: Log is configured to *not* fsync() on all Append() calls
I20260620 00:07:04.666843 5473 tablet_bootstrap.cc:492] T c0bef085db6241b1b81052bcf7c7f9fa P ba1d36c3a4f647b2a04a8b0f895d6904: Bootstrap starting.
I20260620 00:07:04.667061 5472 tablet_bootstrap.cc:492] T c0bef085db6241b1b81052bcf7c7f9fa P 695f03e59fdb4c64a83cb81e06d01b6a: No bootstrap required, opened a new log
I20260620 00:07:04.667191 5474 tablet_bootstrap.cc:492] T c0bef085db6241b1b81052bcf7c7f9fa P 8853c2e8c61e42889a609fcd895a6eb6: No bootstrap required, opened a new log
I20260620 00:07:04.667208 5472 ts_tablet_manager.cc:1397] T c0bef085db6241b1b81052bcf7c7f9fa P 695f03e59fdb4c64a83cb81e06d01b6a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:07:04.667259 5474 ts_tablet_manager.cc:1397] T c0bef085db6241b1b81052bcf7c7f9fa P 8853c2e8c61e42889a609fcd895a6eb6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:07:04.667496 5473 tablet_bootstrap.cc:654] T c0bef085db6241b1b81052bcf7c7f9fa P ba1d36c3a4f647b2a04a8b0f895d6904: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:04.667784 5473 log.cc:824] T c0bef085db6241b1b81052bcf7c7f9fa P ba1d36c3a4f647b2a04a8b0f895d6904: Log is configured to *not* fsync() on all Append() calls
I20260620 00:07:04.668470 5473 tablet_bootstrap.cc:492] T c0bef085db6241b1b81052bcf7c7f9fa P ba1d36c3a4f647b2a04a8b0f895d6904: No bootstrap required, opened a new log
I20260620 00:07:04.668551 5473 ts_tablet_manager.cc:1397] T c0bef085db6241b1b81052bcf7c7f9fa P ba1d36c3a4f647b2a04a8b0f895d6904: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:07:04.669972 5474 raft_consensus.cc:348] T c0bef085db6241b1b81052bcf7c7f9fa P 8853c2e8c61e42889a609fcd895a6eb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1d36c3a4f647b2a04a8b0f895d6904" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45901 } } peers { permanent_uuid: "695f03e59fdb4c64a83cb81e06d01b6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40779 } } peers { permanent_uuid: "8853c2e8c61e42889a609fcd895a6eb6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35501 } }
I20260620 00:07:04.669972 5472 raft_consensus.cc:348] T c0bef085db6241b1b81052bcf7c7f9fa P 695f03e59fdb4c64a83cb81e06d01b6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1d36c3a4f647b2a04a8b0f895d6904" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45901 } } peers { permanent_uuid: "695f03e59fdb4c64a83cb81e06d01b6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40779 } } peers { permanent_uuid: "8853c2e8c61e42889a609fcd895a6eb6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35501 } }
I20260620 00:07:04.670152 5474 raft_consensus.cc:374] T c0bef085db6241b1b81052bcf7c7f9fa P 8853c2e8c61e42889a609fcd895a6eb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:04.670159 5472 raft_consensus.cc:374] T c0bef085db6241b1b81052bcf7c7f9fa P 695f03e59fdb4c64a83cb81e06d01b6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:04.670193 5472 raft_consensus.cc:729] T c0bef085db6241b1b81052bcf7c7f9fa P 695f03e59fdb4c64a83cb81e06d01b6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 695f03e59fdb4c64a83cb81e06d01b6a, State: Initialized, Role: FOLLOWER
I20260620 00:07:04.670193 5474 raft_consensus.cc:729] T c0bef085db6241b1b81052bcf7c7f9fa P 8853c2e8c61e42889a609fcd895a6eb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8853c2e8c61e42889a609fcd895a6eb6, State: Initialized, Role: FOLLOWER
I20260620 00:07:04.670326 5474 consensus_queue.cc:260] T c0bef085db6241b1b81052bcf7c7f9fa P 8853c2e8c61e42889a609fcd895a6eb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1d36c3a4f647b2a04a8b0f895d6904" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45901 } } peers { permanent_uuid: "695f03e59fdb4c64a83cb81e06d01b6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40779 } } peers { permanent_uuid: "8853c2e8c61e42889a609fcd895a6eb6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35501 } }
I20260620 00:07:04.670315 5472 consensus_queue.cc:260] T c0bef085db6241b1b81052bcf7c7f9fa P 695f03e59fdb4c64a83cb81e06d01b6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1d36c3a4f647b2a04a8b0f895d6904" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45901 } } peers { permanent_uuid: "695f03e59fdb4c64a83cb81e06d01b6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40779 } } peers { permanent_uuid: "8853c2e8c61e42889a609fcd895a6eb6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35501 } }
I20260620 00:07:04.670580 5472 ts_tablet_manager.cc:1428] T c0bef085db6241b1b81052bcf7c7f9fa P 695f03e59fdb4c64a83cb81e06d01b6a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:07:04.670580 5474 ts_tablet_manager.cc:1428] T c0bef085db6241b1b81052bcf7c7f9fa P 8853c2e8c61e42889a609fcd895a6eb6: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260620 00:07:04.670701 5196 heartbeater.cc:503] Master 127.0.98.62:36249 was elected leader, sending a full tablet report...
I20260620 00:07:04.670658 5473 raft_consensus.cc:348] T c0bef085db6241b1b81052bcf7c7f9fa P ba1d36c3a4f647b2a04a8b0f895d6904 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1d36c3a4f647b2a04a8b0f895d6904" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45901 } } peers { permanent_uuid: "695f03e59fdb4c64a83cb81e06d01b6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40779 } } peers { permanent_uuid: "8853c2e8c61e42889a609fcd895a6eb6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35501 } }
I20260620 00:07:04.670806 5473 raft_consensus.cc:374] T c0bef085db6241b1b81052bcf7c7f9fa P ba1d36c3a4f647b2a04a8b0f895d6904 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:04.670840 5473 raft_consensus.cc:729] T c0bef085db6241b1b81052bcf7c7f9fa P ba1d36c3a4f647b2a04a8b0f895d6904 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba1d36c3a4f647b2a04a8b0f895d6904, State: Initialized, Role: FOLLOWER
I20260620 00:07:04.670960 5473 consensus_queue.cc:260] T c0bef085db6241b1b81052bcf7c7f9fa P ba1d36c3a4f647b2a04a8b0f895d6904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1d36c3a4f647b2a04a8b0f895d6904" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45901 } } peers { permanent_uuid: "695f03e59fdb4c64a83cb81e06d01b6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40779 } } peers { permanent_uuid: "8853c2e8c61e42889a609fcd895a6eb6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35501 } }
I20260620 00:07:04.671053 5327 heartbeater.cc:503] Master 127.0.98.62:36249 was elected leader, sending a full tablet report...
I20260620 00:07:04.671236 5473 ts_tablet_manager.cc:1428] T c0bef085db6241b1b81052bcf7c7f9fa P ba1d36c3a4f647b2a04a8b0f895d6904: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:07:04.671474 5458 heartbeater.cc:503] Master 127.0.98.62:36249 was elected leader, sending a full tablet report...
W20260620 00:07:04.763720 5328 tablet.cc:2369] T c0bef085db6241b1b81052bcf7c7f9fa P 8853c2e8c61e42889a609fcd895a6eb6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:07:04.844004 5479 raft_consensus.cc:482] T c0bef085db6241b1b81052bcf7c7f9fa P 8853c2e8c61e42889a609fcd895a6eb6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:07:04.844195 5479 raft_consensus.cc:504] T c0bef085db6241b1b81052bcf7c7f9fa P 8853c2e8c61e42889a609fcd895a6eb6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1d36c3a4f647b2a04a8b0f895d6904" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45901 } } peers { permanent_uuid: "695f03e59fdb4c64a83cb81e06d01b6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40779 } } peers { permanent_uuid: "8853c2e8c61e42889a609fcd895a6eb6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35501 } }
I20260620 00:07:04.844709 5479 leader_election.cc:288] T c0bef085db6241b1b81052bcf7c7f9fa P 8853c2e8c61e42889a609fcd895a6eb6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba1d36c3a4f647b2a04a8b0f895d6904 (127.0.98.3:45901), 695f03e59fdb4c64a83cb81e06d01b6a (127.0.98.1:40779)
I20260620 00:07:04.846135 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:04.845066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38049 (local address 127.0.98.3:45901)
0620 00:07:04.845117 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:04.845119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:04.845130 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:07:04.845167 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:04.845168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:04.845172 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:04.845223 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:04.845876 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:04.845878 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:04.845878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:07:04.845884 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:04.845903 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:07:04.845908 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:04.845993 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:04.846053 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 00:07:04.846068 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 00:07:04.846135 5329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:04.844946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45901 (local address 127.0.98.2:38049)
0620 00:07:04.845013 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:04.845027 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:04.845086 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:04.845260 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:04.845261 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:04.845264 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:04.845766 (+ 502us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:07:04.845766 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:04.845773 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:07:04.845785 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:04.845993 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:04.845993 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:04.845994 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 00:07:04.846051 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 00:07:04.846068 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 00:07:04.846345 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:04.845042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43771 (local address 127.0.98.1:40779)
0620 00:07:04.845096 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:04.845098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:04.845277 (+ 179us) server_negotiation.cc:407] Connection header received
0620 00:07:04.845311 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:04.845312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:04.845317 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:04.845436 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:04.846039 (+ 603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:04.846040 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:04.846040 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:07:04.846045 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:04.846069 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:07:04.846073 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:04.846141 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:04.846256 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 00:07:04.846274 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 00:07:04.846520 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:04.844946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40779 (local address 127.0.98.2:43771)
0620 00:07:04.845180 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:04.845188 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:04.845265 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:04.845448 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:04.845449 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:04.845452 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:04.845922 (+ 470us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:07:04.845923 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:04.845930 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:07:04.845952 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:04.846171 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:04.846172 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:04.846173 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 00:07:04.846428 (+ 255us) client_negotiation.cc:240] Negotiation successful
0620 00:07:04.846443 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":59,"thread_start_us":93,"threads_started":1}
I20260620 00:07:04.846498 5412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0bef085db6241b1b81052bcf7c7f9fa" candidate_uuid: "8853c2e8c61e42889a609fcd895a6eb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba1d36c3a4f647b2a04a8b0f895d6904" is_pre_election: true
I20260620 00:07:04.846683 5412 raft_consensus.cc:2454] T c0bef085db6241b1b81052bcf7c7f9fa P ba1d36c3a4f647b2a04a8b0f895d6904 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8853c2e8c61e42889a609fcd895a6eb6 in term 0.
I20260620 00:07:04.846874 5150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0bef085db6241b1b81052bcf7c7f9fa" candidate_uuid: "8853c2e8c61e42889a609fcd895a6eb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "695f03e59fdb4c64a83cb81e06d01b6a" is_pre_election: true
I20260620 00:07:04.846948 5216 leader_election.cc:302] T c0bef085db6241b1b81052bcf7c7f9fa P 8853c2e8c61e42889a609fcd895a6eb6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8853c2e8c61e42889a609fcd895a6eb6, ba1d36c3a4f647b2a04a8b0f895d6904; no voters:
I20260620 00:07:04.847028 5150 raft_consensus.cc:2454] T c0bef085db6241b1b81052bcf7c7f9fa P 695f03e59fdb4c64a83cb81e06d01b6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8853c2e8c61e42889a609fcd895a6eb6 in term 0.
I20260620 00:07:04.847075 5479 raft_consensus.cc:2781] T c0bef085db6241b1b81052bcf7c7f9fa P 8853c2e8c61e42889a609fcd895a6eb6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:07:04.847128 5479 raft_consensus.cc:482] T c0bef085db6241b1b81052bcf7c7f9fa P 8853c2e8c61e42889a609fcd895a6eb6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:07:04.847155 5479 raft_consensus.cc:3037] T c0bef085db6241b1b81052bcf7c7f9fa P 8853c2e8c61e42889a609fcd895a6eb6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:04.847895 5479 raft_consensus.cc:504] T c0bef085db6241b1b81052bcf7c7f9fa P 8853c2e8c61e42889a609fcd895a6eb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1d36c3a4f647b2a04a8b0f895d6904" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45901 } } peers { permanent_uuid: "695f03e59fdb4c64a83cb81e06d01b6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40779 } } peers { permanent_uuid: "8853c2e8c61e42889a609fcd895a6eb6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35501 } }
I20260620 00:07:04.848052 5479 leader_election.cc:288] T c0bef085db6241b1b81052bcf7c7f9fa P 8853c2e8c61e42889a609fcd895a6eb6 [CANDIDATE]: Term 1 election: Requested vote from peers ba1d36c3a4f647b2a04a8b0f895d6904 (127.0.98.3:45901), 695f03e59fdb4c64a83cb81e06d01b6a (127.0.98.1:40779)
I20260620 00:07:04.848208 5412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0bef085db6241b1b81052bcf7c7f9fa" candidate_uuid: "8853c2e8c61e42889a609fcd895a6eb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba1d36c3a4f647b2a04a8b0f895d6904"
I20260620 00:07:04.848208 5150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0bef085db6241b1b81052bcf7c7f9fa" candidate_uuid: "8853c2e8c61e42889a609fcd895a6eb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "695f03e59fdb4c64a83cb81e06d01b6a"
I20260620 00:07:04.848302 5150 raft_consensus.cc:3037] T c0bef085db6241b1b81052bcf7c7f9fa P 695f03e59fdb4c64a83cb81e06d01b6a [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:04.848304 5412 raft_consensus.cc:3037] T c0bef085db6241b1b81052bcf7c7f9fa P ba1d36c3a4f647b2a04a8b0f895d6904 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:04.849238 5412 raft_consensus.cc:2454] T c0bef085db6241b1b81052bcf7c7f9fa P ba1d36c3a4f647b2a04a8b0f895d6904 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8853c2e8c61e42889a609fcd895a6eb6 in term 1.
I20260620 00:07:04.849238 5150 raft_consensus.cc:2454] T c0bef085db6241b1b81052bcf7c7f9fa P 695f03e59fdb4c64a83cb81e06d01b6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8853c2e8c61e42889a609fcd895a6eb6 in term 1.
I20260620 00:07:04.849475 5217 leader_election.cc:302] T c0bef085db6241b1b81052bcf7c7f9fa P 8853c2e8c61e42889a609fcd895a6eb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 695f03e59fdb4c64a83cb81e06d01b6a, 8853c2e8c61e42889a609fcd895a6eb6; no voters:
I20260620 00:07:04.849608 5479 raft_consensus.cc:2781] T c0bef085db6241b1b81052bcf7c7f9fa P 8853c2e8c61e42889a609fcd895a6eb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:07:04.849781 5479 raft_consensus.cc:686] T c0bef085db6241b1b81052bcf7c7f9fa P 8853c2e8c61e42889a609fcd895a6eb6 [term 1 LEADER]: Becoming Leader. State: Replica: 8853c2e8c61e42889a609fcd895a6eb6, State: Running, Role: LEADER
I20260620 00:07:04.849891 5479 consensus_queue.cc:237] T c0bef085db6241b1b81052bcf7c7f9fa P 8853c2e8c61e42889a609fcd895a6eb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1d36c3a4f647b2a04a8b0f895d6904" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45901 } } peers { permanent_uuid: "695f03e59fdb4c64a83cb81e06d01b6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40779 } } peers { permanent_uuid: "8853c2e8c61e42889a609fcd895a6eb6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35501 } }
I20260620 00:07:04.850864 5006 catalog_manager.cc:5495] T c0bef085db6241b1b81052bcf7c7f9fa P 8853c2e8c61e42889a609fcd895a6eb6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8853c2e8c61e42889a609fcd895a6eb6 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "8853c2e8c61e42889a609fcd895a6eb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1d36c3a4f647b2a04a8b0f895d6904" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "695f03e59fdb4c64a83cb81e06d01b6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8853c2e8c61e42889a609fcd895a6eb6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35501 } health_report { overall_health: HEALTHY } } }
I20260620 00:07:04.888784 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:04.887798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33178 (local address 127.0.98.2:35501)
0620 00:07:04.887869 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:04.887870 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:04.887878 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:07:04.887912 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:04.887913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:04.887926 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:04.887974 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:04.888584 (+ 610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:04.888585 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:04.888585 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:07:04.888588 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:04.888600 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:07:04.888603 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:04.888648 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:04.888717 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 00:07:04.888730 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 00:07:04.890755 5197 tablet.cc:2369] T c0bef085db6241b1b81052bcf7c7f9fa P 695f03e59fdb4c64a83cb81e06d01b6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:07:04.890841 5412 raft_consensus.cc:1261] T c0bef085db6241b1b81052bcf7c7f9fa P ba1d36c3a4f647b2a04a8b0f895d6904 [term 1 FOLLOWER]: Refusing update from remote peer 8853c2e8c61e42889a609fcd895a6eb6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:07:04.891023 5150 raft_consensus.cc:1261] T c0bef085db6241b1b81052bcf7c7f9fa P 695f03e59fdb4c64a83cb81e06d01b6a [term 1 FOLLOWER]: Refusing update from remote peer 8853c2e8c61e42889a609fcd895a6eb6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:07:04.891237 5482 consensus_queue.cc:1035] T c0bef085db6241b1b81052bcf7c7f9fa P 8853c2e8c61e42889a609fcd895a6eb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba1d36c3a4f647b2a04a8b0f895d6904" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:07:04.891352 5479 consensus_queue.cc:1035] T c0bef085db6241b1b81052bcf7c7f9fa P 8853c2e8c61e42889a609fcd895a6eb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "695f03e59fdb4c64a83cb81e06d01b6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 00:07:04.891505 5459 tablet.cc:2369] T c0bef085db6241b1b81052bcf7c7f9fa P ba1d36c3a4f647b2a04a8b0f895d6904: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:07:04.894263 5485 mvcc.cc:205] Tried to move back new op lower bound from 7298719845949075456 to 7298719845785726976. Current Snapshot: MvccSnapshot[applied={T|T < 7298719845949075456}]
I20260620 00:07:04.894836 5486 mvcc.cc:205] Tried to move back new op lower bound from 7298719845949075456 to 7298719845785726976. Current Snapshot: MvccSnapshot[applied={T|T < 7298719845949075456}]
I20260620 00:07:04.896546 5008 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:07:04.896673 5008 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:07:04.897434 5008 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2e515fb2e8b841f7a50151e9e7356ddd: Sending DeleteTablet for 3 replicas of tablet c0bef085db6241b1b81052bcf7c7f9fa
I20260620 00:07:04.897779 5392 tablet_service.cc:1514] Processing DeleteTablet for tablet c0bef085db6241b1b81052bcf7c7f9fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:07:04 UTC) from {username='slave'} at 127.0.0.1:33266
I20260620 00:07:04.897878 5130 tablet_service.cc:1514] Processing DeleteTablet for tablet c0bef085db6241b1b81052bcf7c7f9fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:07:04 UTC) from {username='slave'} at 127.0.0.1:34482
I20260620 00:07:04.898074 5497 tablet_replica.cc:330] T c0bef085db6241b1b81052bcf7c7f9fa P 695f03e59fdb4c64a83cb81e06d01b6a: stopping tablet replica
I20260620 00:07:04.898061 5261 tablet_service.cc:1514] Processing DeleteTablet for tablet c0bef085db6241b1b81052bcf7c7f9fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:07:04 UTC) from {username='slave'} at 127.0.0.1:33170
I20260620 00:07:04.898074 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 5068
I20260620 00:07:04.898202 5497 raft_consensus.cc:2229] T c0bef085db6241b1b81052bcf7c7f9fa P 695f03e59fdb4c64a83cb81e06d01b6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:07:04.898313 5497 raft_consensus.cc:2258] T c0bef085db6241b1b81052bcf7c7f9fa P 695f03e59fdb4c64a83cb81e06d01b6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:07:04.898440 5498 tablet_replica.cc:330] T c0bef085db6241b1b81052bcf7c7f9fa P 8853c2e8c61e42889a609fcd895a6eb6: stopping tablet replica
I20260620 00:07:04.898572 5498 raft_consensus.cc:2229] T c0bef085db6241b1b81052bcf7c7f9fa P 8853c2e8c61e42889a609fcd895a6eb6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:07:04.898733 5498 raft_consensus.cc:2258] T c0bef085db6241b1b81052bcf7c7f9fa P 8853c2e8c61e42889a609fcd895a6eb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:07:04.899261 5498 ts_tablet_manager.cc:1905] T c0bef085db6241b1b81052bcf7c7f9fa P 8853c2e8c61e42889a609fcd895a6eb6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:07:04.900988 5498 ts_tablet_manager.cc:1918] T c0bef085db6241b1b81052bcf7c7f9fa P 8853c2e8c61e42889a609fcd895a6eb6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:07:04.901113 5498 log.cc:1196] T c0bef085db6241b1b81052bcf7c7f9fa P 8853c2e8c61e42889a609fcd895a6eb6: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-1/wal/wals/c0bef085db6241b1b81052bcf7c7f9fa
I20260620 00:07:04.901468 5496 tablet_replica.cc:330] T c0bef085db6241b1b81052bcf7c7f9fa P ba1d36c3a4f647b2a04a8b0f895d6904: stopping tablet replica
I20260620 00:07:04.901504 5498 ts_tablet_manager.cc:1939] T c0bef085db6241b1b81052bcf7c7f9fa P 8853c2e8c61e42889a609fcd895a6eb6: Deleting consensus metadata
I20260620 00:07:04.901589 5496 raft_consensus.cc:2229] T c0bef085db6241b1b81052bcf7c7f9fa P ba1d36c3a4f647b2a04a8b0f895d6904 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:07:04.901696 5496 raft_consensus.cc:2258] T c0bef085db6241b1b81052bcf7c7f9fa P ba1d36c3a4f647b2a04a8b0f895d6904 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:07:04.902089 4995 catalog_manager.cc:4841] TS 8853c2e8c61e42889a609fcd895a6eb6 (127.0.98.2:35501): tablet c0bef085db6241b1b81052bcf7c7f9fa (table test-table [id=267c619075da45f49afb7233d7b18338]) successfully deleted
I20260620 00:07:04.902153 5496 ts_tablet_manager.cc:1905] T c0bef085db6241b1b81052bcf7c7f9fa P ba1d36c3a4f647b2a04a8b0f895d6904: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:07:04.904031 5496 ts_tablet_manager.cc:1918] T c0bef085db6241b1b81052bcf7c7f9fa P ba1d36c3a4f647b2a04a8b0f895d6904: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:07:04.904110 5496 log.cc:1196] T c0bef085db6241b1b81052bcf7c7f9fa P ba1d36c3a4f647b2a04a8b0f895d6904: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914001714546-392-0/minicluster-data/ts-2/wal/wals/c0bef085db6241b1b81052bcf7c7f9fa
I20260620 00:07:04.904374 5496 ts_tablet_manager.cc:1939] T c0bef085db6241b1b81052bcf7c7f9fa P ba1d36c3a4f647b2a04a8b0f895d6904: Deleting consensus metadata
I20260620 00:07:04.905714 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 5199
I20260620 00:07:04.905839 4994 catalog_manager.cc:4841] TS ba1d36c3a4f647b2a04a8b0f895d6904 (127.0.98.3:45901): tablet c0bef085db6241b1b81052bcf7c7f9fa (table test-table [id=267c619075da45f49afb7233d7b18338]) successfully deleted
W20260620 00:07:04.906630 4995 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40779 (error 108)
I20260620 00:07:04.907173 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:04.907032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40779 (local address 127.0.0.1:34492)
0620 00:07:04.907094 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:04.907139 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 00:07:04.907379 4995 catalog_manager.cc:4584] TS 695f03e59fdb4c64a83cb81e06d01b6a (127.0.98.1:40779): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c0bef085db6241b1b81052bcf7c7f9fa: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40779: connect: Connection refused (error 111)
I20260620 00:07:04.910696 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 5330
I20260620 00:07:04.914953 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 4977
2026-06-20T00:07:04Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (830 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 00:07:04.935429 392 mini_oidc.cc:152] Starting JWKS server
I20260620 00:07:04.936502 392 webserver.cc:460] Webserver started at https://127.0.0.1:40457/ using document root <none> and password file <none>
I20260620 00:07:04.936623 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 00:07:04.936810 392 webserver.cc:460] Webserver started at http://127.0.0.1:43533/ using document root <none> and password file <none>
2026-06-20T00:07:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:07:04Z Disabled control of system clock
I20260620 00:07:04.944644 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38795
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38795
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40457/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914001714546-392-0/testchainca.cert with env {}
W20260620 00:07:05.027395 5505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:05.027603 5505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:05.027627 5505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:05.028822 5505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:07:05.028867 5505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:05.028879 5505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:05.028894 5505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:07:05.028950 5505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 00:07:05.028967 5505 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 00:07:05.028991 5505 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40457/jwks.json
I20260620 00:07:05.030581 5505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38795
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40457/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914001714546-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:05.030854 5505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:05.031097 5505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:05.036468 5510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:05.036497 5513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:05.036628 5511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:05.036494 5505 server_base.cc:1034] running on GCE node
I20260620 00:07:05.036929 5505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:05.037309 5505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:05.038558 5505 hybrid_clock.cc:648] HybridClock initialized: now 1781914025038547 us; error 30 us; skew 500 ppm
I20260620 00:07:05.039898 5505 webserver.cc:460] Webserver started at http://127.0.98.62:42441/ using document root <none> and password file <none>
I20260620 00:07:05.040131 5505 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:05.040177 5505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:05.040267 5505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:05.041293 5505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "5aebb477cddc47c892d00447eae15e0d"
format_stamp: "Formatted at 2026-06-20 00:07:05 on dist-test-slave-grqt"
server_key: "864b93b4d476108ae808a3a26aad4a32"
server_key_iv: "dfb8a36221e6308cca105dec9acbc020"
server_key_version: "encryptionkey@0"
I20260620 00:07:05.041695 5505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "5aebb477cddc47c892d00447eae15e0d"
format_stamp: "Formatted at 2026-06-20 00:07:05 on dist-test-slave-grqt"
server_key: "864b93b4d476108ae808a3a26aad4a32"
server_key_iv: "dfb8a36221e6308cca105dec9acbc020"
server_key_version: "encryptionkey@0"
I20260620 00:07:05.042982 5505 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 00:07:05.043897 5519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:05.044139 5505 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:05.044252 5505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "5aebb477cddc47c892d00447eae15e0d"
format_stamp: "Formatted at 2026-06-20 00:07:05 on dist-test-slave-grqt"
server_key: "864b93b4d476108ae808a3a26aad4a32"
server_key_iv: "dfb8a36221e6308cca105dec9acbc020"
server_key_version: "encryptionkey@0"
I20260620 00:07:05.044349 5505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:05.056838 5505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:05.057189 5505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:05.057349 5505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:05.061111 5505 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38795
I20260620 00:07:05.061192 5571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38795 every 8 connection(s)
I20260620 00:07:05.061610 5505 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:07:05.062629 5572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:05.064908 5572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aebb477cddc47c892d00447eae15e0d: Bootstrap starting.
I20260620 00:07:05.065577 5572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5aebb477cddc47c892d00447eae15e0d: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:05.065878 5572 log.cc:824] T 00000000000000000000000000000000 P 5aebb477cddc47c892d00447eae15e0d: Log is configured to *not* fsync() on all Append() calls
I20260620 00:07:05.066598 5572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aebb477cddc47c892d00447eae15e0d: No bootstrap required, opened a new log
I20260620 00:07:05.068136 5572 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5aebb477cddc47c892d00447eae15e0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aebb477cddc47c892d00447eae15e0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38795 } }
I20260620 00:07:05.068312 5572 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5aebb477cddc47c892d00447eae15e0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:05.068336 5572 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5aebb477cddc47c892d00447eae15e0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aebb477cddc47c892d00447eae15e0d, State: Initialized, Role: FOLLOWER
I20260620 00:07:05.068512 5572 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5aebb477cddc47c892d00447eae15e0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aebb477cddc47c892d00447eae15e0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38795 } }
I20260620 00:07:05.068576 5572 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5aebb477cddc47c892d00447eae15e0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:07:05.068596 5572 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5aebb477cddc47c892d00447eae15e0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:07:05.068665 5572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5aebb477cddc47c892d00447eae15e0d [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:05.069090 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 5505
I20260620 00:07:05.069231 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:07:05.069252 5572 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5aebb477cddc47c892d00447eae15e0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aebb477cddc47c892d00447eae15e0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38795 } }
I20260620 00:07:05.069398 5572 leader_election.cc:302] T 00000000000000000000000000000000 P 5aebb477cddc47c892d00447eae15e0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5aebb477cddc47c892d00447eae15e0d; no voters:
I20260620 00:07:05.069446 392 external_mini_cluster.cc:1428] Setting key ac61b99efe5c3aa0c222898840876018
I20260620 00:07:05.069586 5572 leader_election.cc:288] T 00000000000000000000000000000000 P 5aebb477cddc47c892d00447eae15e0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:07:05.069643 5575 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5aebb477cddc47c892d00447eae15e0d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:07:05.069772 5575 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5aebb477cddc47c892d00447eae15e0d [term 1 LEADER]: Becoming Leader. State: Replica: 5aebb477cddc47c892d00447eae15e0d, State: Running, Role: LEADER
I20260620 00:07:05.069850 5572 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5aebb477cddc47c892d00447eae15e0d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:07:05.069865 5575 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5aebb477cddc47c892d00447eae15e0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aebb477cddc47c892d00447eae15e0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38795 } }
I20260620 00:07:05.070168 5575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aebb477cddc47c892d00447eae15e0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5aebb477cddc47c892d00447eae15e0d. Latest consensus state: current_term: 1 leader_uuid: "5aebb477cddc47c892d00447eae15e0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aebb477cddc47c892d00447eae15e0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38795 } } }
I20260620 00:07:05.070241 5575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aebb477cddc47c892d00447eae15e0d [sys.catalog]: This master's current role is: LEADER
I20260620 00:07:05.070426 5576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aebb477cddc47c892d00447eae15e0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5aebb477cddc47c892d00447eae15e0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aebb477cddc47c892d00447eae15e0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38795 } } }
I20260620 00:07:05.070537 5576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aebb477cddc47c892d00447eae15e0d [sys.catalog]: This master's current role is: LEADER
I20260620 00:07:05.071171 5582 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:07:05.071758 5582 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:07:05.074018 5582 catalog_manager.cc:1347] Generated new cluster ID: 3e68deb34c21470ebd0b188e4a06c0ba
I20260620 00:07:05.074074 5582 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:07:05.076264 5580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:05.070084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55510 (local address 127.0.98.62:38795)
0620 00:07:05.070322 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:05.070332 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:05.070778 (+ 446us) server_negotiation.cc:407] Connection header received
0620 00:07:05.070917 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:05.070926 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:05.071008 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:05.071175 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:05.071637 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:05.072910 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:05.073966 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:05.074575 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:05.074687 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:05.075557 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:05.075561 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:05.075561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:07:05.075759 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:05.075797 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:07:05.075810 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:05.075845 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:05.075971 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 00:07:05.076049 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":72,"threads_started":1}
I20260620 00:07:05.077303 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:07:05.093884 5582 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:07:05.094623 5582 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:07:05.106853 5582 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5aebb477cddc47c892d00447eae15e0d: Generated new TSK 0
I20260620 00:07:05.107081 5582 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 00:07:05.141947 5602 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 00:07:05.142066 5602 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 00:07:05.142333 5580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:05.119898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55526 (local address 127.0.98.62:38795)
0620 00:07:05.119971 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:05.119973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:05.120123 (+ 150us) server_negotiation.cc:407] Connection header received
0620 00:07:05.120260 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:05.120261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:05.120275 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:05.120359 (+ 84us) server_negotiation.cc:226] Negotiated authn=JWT
0620 00:07:05.120523 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:05.121024 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:05.121907 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:05.122116 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:05.122208 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:05.122441 (+ 233us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 00:07:05.141943 (+ 19502us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40457/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 00:07:05.142247 (+ 304us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55526: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40457/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 00:07:05.142395 5580 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55526: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40457/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 00:07:05.142616 5601 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:05.119833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38795 (local address 127.0.0.1:55526)
0620 00:07:05.120042 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:05.120051 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:05.120145 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:05.120368 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:05.120369 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:05.120371 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 00:07:05.120443 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:05.120446 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:05.121126 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:05.121127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:05.121818 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:05.121820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:05.122215 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:05.122218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:05.122284 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:05.122306 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 00:07:05.142173 (+ 19867us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40457/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 00:07:05.142297 (+ 124us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:38795: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40457/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":106,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:38795: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40457/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 00:07:05.143066 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 5505
2026-06-20T00:07:05Z chronyd exiting
I20260620 00:07:05.179378 392 test_util.cc:175] -----------------------------------------------
I20260620 00:07:05.179466 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914001714546-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (245 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 00:07:05.180225 392 mini_oidc.cc:152] Starting JWKS server
I20260620 00:07:05.181133 392 webserver.cc:460] Webserver started at https://127.0.0.1:41919/ using document root <none> and password file <none>
I20260620 00:07:05.181185 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 00:07:05.181358 392 webserver.cc:460] Webserver started at http://127.0.0.1:33475/ using document root <none> and password file <none>
2026-06-20T00:07:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:07:05Z Disabled control of system clock
I20260620 00:07:05.189164 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34191
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34191
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41919/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914001714546-392-0/testchainca.cert with env {}
W20260620 00:07:05.271288 5608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:05.271551 5608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:05.271588 5608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:05.272776 5608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:07:05.272853 5608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:05.272876 5608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:05.272905 5608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:07:05.272935 5608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 00:07:05.272958 5608 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 00:07:05.272986 5608 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41919/jwks.json
I20260620 00:07:05.274556 5608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34191
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41919/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914001714546-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:05.274847 5608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:05.275087 5608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:05.280180 5614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:05.280213 5613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:05.280231 5608 server_base.cc:1034] running on GCE node
W20260620 00:07:05.280262 5616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:05.280658 5608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:05.280974 5608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:05.282135 5608 hybrid_clock.cc:648] HybridClock initialized: now 1781914025282124 us; error 34 us; skew 500 ppm
I20260620 00:07:05.283519 5608 webserver.cc:460] Webserver started at http://127.0.98.62:34727/ using document root <none> and password file <none>
I20260620 00:07:05.283774 5608 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:05.283821 5608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:05.283960 5608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:05.285066 5608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "f984063568f343cca636cfc0f634c6bb"
format_stamp: "Formatted at 2026-06-20 00:07:05 on dist-test-slave-grqt"
server_key: "3c815e6859d16fdc06c83aa8f7eb8959"
server_key_iv: "e1ee74ae35a7de2cba7d984154bf3713"
server_key_version: "encryptionkey@0"
I20260620 00:07:05.285521 5608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "f984063568f343cca636cfc0f634c6bb"
format_stamp: "Formatted at 2026-06-20 00:07:05 on dist-test-slave-grqt"
server_key: "3c815e6859d16fdc06c83aa8f7eb8959"
server_key_iv: "e1ee74ae35a7de2cba7d984154bf3713"
server_key_version: "encryptionkey@0"
I20260620 00:07:05.286965 5608 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:05.287777 5622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:05.287966 5608 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:07:05.288067 5608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "f984063568f343cca636cfc0f634c6bb"
format_stamp: "Formatted at 2026-06-20 00:07:05 on dist-test-slave-grqt"
server_key: "3c815e6859d16fdc06c83aa8f7eb8959"
server_key_iv: "e1ee74ae35a7de2cba7d984154bf3713"
server_key_version: "encryptionkey@0"
I20260620 00:07:05.288156 5608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:05.319437 5608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:05.319793 5608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:05.319952 5608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:05.323923 5608 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34191
I20260620 00:07:05.323974 5674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34191 every 8 connection(s)
I20260620 00:07:05.324352 5608 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:07:05.325485 5675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:05.327752 5675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f984063568f343cca636cfc0f634c6bb: Bootstrap starting.
I20260620 00:07:05.328361 5675 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f984063568f343cca636cfc0f634c6bb: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:05.328648 5675 log.cc:824] T 00000000000000000000000000000000 P f984063568f343cca636cfc0f634c6bb: Log is configured to *not* fsync() on all Append() calls
I20260620 00:07:05.329356 5675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f984063568f343cca636cfc0f634c6bb: No bootstrap required, opened a new log
I20260620 00:07:05.330945 5675 raft_consensus.cc:348] T 00000000000000000000000000000000 P f984063568f343cca636cfc0f634c6bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f984063568f343cca636cfc0f634c6bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34191 } }
I20260620 00:07:05.331090 5675 raft_consensus.cc:374] T 00000000000000000000000000000000 P f984063568f343cca636cfc0f634c6bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:05.331121 5675 raft_consensus.cc:729] T 00000000000000000000000000000000 P f984063568f343cca636cfc0f634c6bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f984063568f343cca636cfc0f634c6bb, State: Initialized, Role: FOLLOWER
I20260620 00:07:05.331297 5675 consensus_queue.cc:260] T 00000000000000000000000000000000 P f984063568f343cca636cfc0f634c6bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f984063568f343cca636cfc0f634c6bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34191 } }
I20260620 00:07:05.331354 5675 raft_consensus.cc:388] T 00000000000000000000000000000000 P f984063568f343cca636cfc0f634c6bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:07:05.331410 5675 raft_consensus.cc:482] T 00000000000000000000000000000000 P f984063568f343cca636cfc0f634c6bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:07:05.331458 5675 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f984063568f343cca636cfc0f634c6bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:05.331991 5675 raft_consensus.cc:504] T 00000000000000000000000000000000 P f984063568f343cca636cfc0f634c6bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f984063568f343cca636cfc0f634c6bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34191 } }
I20260620 00:07:05.332122 5675 leader_election.cc:302] T 00000000000000000000000000000000 P f984063568f343cca636cfc0f634c6bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f984063568f343cca636cfc0f634c6bb; no voters:
I20260620 00:07:05.332314 5675 leader_election.cc:288] T 00000000000000000000000000000000 P f984063568f343cca636cfc0f634c6bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:07:05.332381 5678 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f984063568f343cca636cfc0f634c6bb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:07:05.332562 5678 raft_consensus.cc:686] T 00000000000000000000000000000000 P f984063568f343cca636cfc0f634c6bb [term 1 LEADER]: Becoming Leader. State: Replica: f984063568f343cca636cfc0f634c6bb, State: Running, Role: LEADER
I20260620 00:07:05.332607 5675 sys_catalog.cc:564] T 00000000000000000000000000000000 P f984063568f343cca636cfc0f634c6bb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:07:05.332664 5678 consensus_queue.cc:237] T 00000000000000000000000000000000 P f984063568f343cca636cfc0f634c6bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f984063568f343cca636cfc0f634c6bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34191 } }
I20260620 00:07:05.333065 5679 sys_catalog.cc:455] T 00000000000000000000000000000000 P f984063568f343cca636cfc0f634c6bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f984063568f343cca636cfc0f634c6bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f984063568f343cca636cfc0f634c6bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34191 } } }
I20260620 00:07:05.333163 5679 sys_catalog.cc:458] T 00000000000000000000000000000000 P f984063568f343cca636cfc0f634c6bb [sys.catalog]: This master's current role is: LEADER
I20260620 00:07:05.333169 5680 sys_catalog.cc:455] T 00000000000000000000000000000000 P f984063568f343cca636cfc0f634c6bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader f984063568f343cca636cfc0f634c6bb. Latest consensus state: current_term: 1 leader_uuid: "f984063568f343cca636cfc0f634c6bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f984063568f343cca636cfc0f634c6bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34191 } } }
I20260620 00:07:05.333256 5680 sys_catalog.cc:458] T 00000000000000000000000000000000 P f984063568f343cca636cfc0f634c6bb [sys.catalog]: This master's current role is: LEADER
I20260620 00:07:05.333408 5687 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:07:05.333962 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 5608
I20260620 00:07:05.334007 5687 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:07:05.334059 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:07:05.334247 392 external_mini_cluster.cc:1428] Setting key 16ab744273fb45f62ce21082ddc1a373
I20260620 00:07:05.335830 5687 catalog_manager.cc:1347] Generated new cluster ID: b8a70894c8c7450e9c1a4f1e4e5df1e5
I20260620 00:07:05.335891 5687 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:07:05.342617 5697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:05.334720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47240 (local address 127.0.98.62:34191)
0620 00:07:05.335123 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:05.335133 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:05.335162 (+ 29us) server_negotiation.cc:407] Connection header received
0620 00:07:05.335321 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:05.335330 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:05.335422 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:05.335571 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:05.337498 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:05.338970 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:05.340140 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:05.340837 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:05.340993 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:05.341795 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:05.341800 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:05.341800 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:07:05.342019 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:05.342067 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:07:05.342084 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:05.342154 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:05.342298 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 00:07:05.342386 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":77,"threads_started":1}
I20260620 00:07:05.342731 5687 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:07:05.343410 5687 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:07:05.343664 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:07:05.347815 5687 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f984063568f343cca636cfc0f634c6bb: Generated new TSK 0
I20260620 00:07:05.348024 5687 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 00:07:05.368795 5705 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 00:07:05.368886 5705 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 00:07:05.369314 5697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:05.354323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47248 (local address 127.0.98.62:34191)
0620 00:07:05.354371 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:05.354373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:05.354481 (+ 108us) server_negotiation.cc:407] Connection header received
0620 00:07:05.354584 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:05.354585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:05.354604 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:05.354684 (+ 80us) server_negotiation.cc:226] Negotiated authn=JWT
0620 00:07:05.354835 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:05.355346 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:05.356185 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:05.356401 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:05.356488 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:05.356697 (+ 209us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 00:07:05.369034 (+ 12337us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41919/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 00:07:05.369221 (+ 187us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47248: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41919/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 00:07:05.369412 5704 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:05.354135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34191 (local address 127.0.0.1:47248)
0620 00:07:05.354396 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:05.354407 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:05.354503 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:05.354695 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:05.354696 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:05.354697 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0620 00:07:05.354756 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:05.354758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:05.355454 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:05.355455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:05.356088 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:05.356090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:05.356490 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:05.356494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:05.356555 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:05.356561 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 00:07:05.369177 (+ 12616us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41919/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 00:07:05.369280 (+ 103us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:34191: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41919/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":148,"threads_started":1}
W20260620 00:07:05.369416 5697 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47248: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41919/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260620 00:07:05.369819 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 5608
2026-06-20T00:07:05Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (227 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 00:07:05.407593 392 mini_oidc.cc:152] Starting JWKS server
I20260620 00:07:05.408270 392 webserver.cc:460] Webserver started at https://127.0.0.1:38699/ using document root <none> and password file <none>
I20260620 00:07:05.408313 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 00:07:05.408478 392 webserver.cc:460] Webserver started at http://127.0.0.1:44371/ using document root <none> and password file <none>
2026-06-20T00:07:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:07:05Z Disabled control of system clock
I20260620 00:07:05.416165 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33709
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33709
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38699/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 00:07:05.496945 5711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:05.497222 5711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:05.497260 5711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:05.498524 5711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:07:05.498603 5711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:05.498624 5711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:05.498644 5711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:07:05.498672 5711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 00:07:05.498692 5711 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 00:07:05.498708 5711 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38699/jwks.json
I20260620 00:07:05.500195 5711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33709
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38699/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:05.500489 5711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:05.500803 5711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:05.505728 5719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:05.505882 5716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:05.505728 5717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:05.505939 5711 server_base.cc:1034] running on GCE node
I20260620 00:07:05.506230 5711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:05.506481 5711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:05.507633 5711 hybrid_clock.cc:648] HybridClock initialized: now 1781914025507609 us; error 38 us; skew 500 ppm
I20260620 00:07:05.508991 5711 webserver.cc:460] Webserver started at http://127.0.98.62:36231/ using document root <none> and password file <none>
I20260620 00:07:05.509279 5711 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:05.509328 5711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:05.509565 5711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:05.510658 5711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "d7a64f394cc440698a0a2912244e38e1"
format_stamp: "Formatted at 2026-06-20 00:07:05 on dist-test-slave-grqt"
server_key: "a8b4e6b6abf8e27b47e9a28a403520c1"
server_key_iv: "ca61f5ad695102a1411a0dda402a2623"
server_key_version: "encryptionkey@0"
I20260620 00:07:05.510964 5711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "d7a64f394cc440698a0a2912244e38e1"
format_stamp: "Formatted at 2026-06-20 00:07:05 on dist-test-slave-grqt"
server_key: "a8b4e6b6abf8e27b47e9a28a403520c1"
server_key_iv: "ca61f5ad695102a1411a0dda402a2623"
server_key_version: "encryptionkey@0"
I20260620 00:07:05.512219 5711 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:07:05.512895 5725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:05.513077 5711 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:07:05.513208 5711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "d7a64f394cc440698a0a2912244e38e1"
format_stamp: "Formatted at 2026-06-20 00:07:05 on dist-test-slave-grqt"
server_key: "a8b4e6b6abf8e27b47e9a28a403520c1"
server_key_iv: "ca61f5ad695102a1411a0dda402a2623"
server_key_version: "encryptionkey@0"
I20260620 00:07:05.513288 5711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:05.547338 5711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:05.547789 5711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:05.547950 5711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:05.552276 5711 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33709
I20260620 00:07:05.552316 5777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33709 every 8 connection(s)
I20260620 00:07:05.552667 5711 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:07:05.553289 5778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:05.556249 5778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7a64f394cc440698a0a2912244e38e1: Bootstrap starting.
I20260620 00:07:05.556870 5778 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7a64f394cc440698a0a2912244e38e1: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:05.557168 5778 log.cc:824] T 00000000000000000000000000000000 P d7a64f394cc440698a0a2912244e38e1: Log is configured to *not* fsync() on all Append() calls
I20260620 00:07:05.557965 5778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7a64f394cc440698a0a2912244e38e1: No bootstrap required, opened a new log
I20260620 00:07:05.559614 5778 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7a64f394cc440698a0a2912244e38e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7a64f394cc440698a0a2912244e38e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } }
I20260620 00:07:05.559798 5778 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7a64f394cc440698a0a2912244e38e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:05.559845 5778 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7a64f394cc440698a0a2912244e38e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7a64f394cc440698a0a2912244e38e1, State: Initialized, Role: FOLLOWER
I20260620 00:07:05.560041 5778 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7a64f394cc440698a0a2912244e38e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7a64f394cc440698a0a2912244e38e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } }
I20260620 00:07:05.560164 5778 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7a64f394cc440698a0a2912244e38e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:07:05.560212 5778 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7a64f394cc440698a0a2912244e38e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:07:05.560266 5778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7a64f394cc440698a0a2912244e38e1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:05.560832 5778 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7a64f394cc440698a0a2912244e38e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7a64f394cc440698a0a2912244e38e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } }
I20260620 00:07:05.560951 5778 leader_election.cc:302] T 00000000000000000000000000000000 P d7a64f394cc440698a0a2912244e38e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7a64f394cc440698a0a2912244e38e1; no voters:
I20260620 00:07:05.560959 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 5711
I20260620 00:07:05.561079 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:07:05.561161 5778 leader_election.cc:288] T 00000000000000000000000000000000 P d7a64f394cc440698a0a2912244e38e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:07:05.561223 5781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7a64f394cc440698a0a2912244e38e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:07:05.561328 392 external_mini_cluster.cc:1428] Setting key 829ecc9c81d2c8516dc388a06a1f0aeb
I20260620 00:07:05.561408 5781 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7a64f394cc440698a0a2912244e38e1 [term 1 LEADER]: Becoming Leader. State: Replica: d7a64f394cc440698a0a2912244e38e1, State: Running, Role: LEADER
I20260620 00:07:05.561479 5778 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7a64f394cc440698a0a2912244e38e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:07:05.561522 5781 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7a64f394cc440698a0a2912244e38e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7a64f394cc440698a0a2912244e38e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } }
I20260620 00:07:05.561892 5781 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7a64f394cc440698a0a2912244e38e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7a64f394cc440698a0a2912244e38e1. Latest consensus state: current_term: 1 leader_uuid: "d7a64f394cc440698a0a2912244e38e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7a64f394cc440698a0a2912244e38e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } } }
I20260620 00:07:05.561976 5781 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7a64f394cc440698a0a2912244e38e1 [sys.catalog]: This master's current role is: LEADER
I20260620 00:07:05.562150 5782 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7a64f394cc440698a0a2912244e38e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7a64f394cc440698a0a2912244e38e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7a64f394cc440698a0a2912244e38e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } } }
I20260620 00:07:05.562294 5782 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7a64f394cc440698a0a2912244e38e1 [sys.catalog]: This master's current role is: LEADER
I20260620 00:07:05.562223 5789 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:07:05.562896 5789 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:07:05.564564 5789 catalog_manager.cc:1347] Generated new cluster ID: 71f100fe1d604cafbe190b343a3ec145
I20260620 00:07:05.564612 5789 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:07:05.567389 5790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:05.562010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50060 (local address 127.0.98.62:33709)
0620 00:07:05.562227 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:05.562237 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:05.562496 (+ 259us) server_negotiation.cc:407] Connection header received
0620 00:07:05.562622 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:05.562628 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:05.562704 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:05.562847 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:05.563215 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:05.564397 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:05.565496 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:05.565908 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:05.566039 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:05.566710 (+ 671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:05.566713 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:05.566713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:07:05.566907 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:05.566939 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:07:05.566950 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:05.567028 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:05.567103 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 00:07:05.567196 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":70,"threads_started":1}
I20260620 00:07:05.568269 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:07:05.593174 5789 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:07:05.593935 5789 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:07:05.604424 5789 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7a64f394cc440698a0a2912244e38e1: Generated new TSK 0
I20260620 00:07:05.604645 5789 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:07:05.622200 5790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:05.618387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50066 (local address 127.0.98.62:33709)
0620 00:07:05.618452 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:05.618454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:05.618601 (+ 147us) server_negotiation.cc:407] Connection header received
0620 00:07:05.618657 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:05.618658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:05.618669 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:05.618741 (+ 72us) server_negotiation.cc:226] Negotiated authn=JWT
0620 00:07:05.618906 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:05.619383 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:05.620528 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:05.620711 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:05.620767 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:05.620966 (+ 199us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 00:07:05.621982 (+ 1016us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38699/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 00:07:05.622115 (+ 133us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50066: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38699/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":36}
W20260620 00:07:05.622272 5790 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50066: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38699/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 00:07:05.622296 5807 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:05.618299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.0.1:50066)
0620 00:07:05.618494 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:05.618506 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:05.618584 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:05.618761 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:05.618762 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:05.618764 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 00:07:05.618832 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:05.618834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:05.619490 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:05.619491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:05.620439 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:05.620442 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:05.620789 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:05.620797 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:05.620864 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:05.620870 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 00:07:05.622084 (+ 1214us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38699/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 00:07:05.622193 (+ 109us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:33709: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38699/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":110,"threads_started":1}
I20260620 00:07:05.622530 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 5711
E20260620 00:07:05.633351 5808 webserver.cc:533] Webserver: sslize failed:
E20260620 00:07:05.633484 5808 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T00:07:05Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (251 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T00:07:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:07:05Z Disabled control of system clock
I20260620 00:07:05.667296 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35381
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35381
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914001714546-392-0/testchainca.cert with env {}
W20260620 00:07:05.748008 5812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:05.748220 5812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:05.748251 5812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:05.749462 5812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:07:05.749503 5812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:05.749516 5812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:05.749528 5812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:07:05.749541 5812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 00:07:05.749599 5812 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 00:07:05.749615 5812 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 00:07:05.751087 5812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35381
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914001714546-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:05.751338 5812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:05.751577 5812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:05.756721 5818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:05.756721 5817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:05.756736 5820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:05.757020 5812 server_base.cc:1034] running on GCE node
I20260620 00:07:05.757279 5812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:05.757587 5812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:05.758742 5812 hybrid_clock.cc:648] HybridClock initialized: now 1781914025758728 us; error 29 us; skew 500 ppm
I20260620 00:07:05.760118 5812 webserver.cc:460] Webserver started at http://127.0.98.62:36247/ using document root <none> and password file <none>
I20260620 00:07:05.760358 5812 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:05.760404 5812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:05.760555 5812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:05.761776 5812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "1e935be8d4634cb5aade3c1f20ce3bd7"
format_stamp: "Formatted at 2026-06-20 00:07:05 on dist-test-slave-grqt"
server_key: "b807abff7bdd38359ec1674117a00ba7"
server_key_iv: "2caea1847ed4b2da6e4e65c0d12d70b7"
server_key_version: "encryptionkey@0"
I20260620 00:07:05.762140 5812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "1e935be8d4634cb5aade3c1f20ce3bd7"
format_stamp: "Formatted at 2026-06-20 00:07:05 on dist-test-slave-grqt"
server_key: "b807abff7bdd38359ec1674117a00ba7"
server_key_iv: "2caea1847ed4b2da6e4e65c0d12d70b7"
server_key_version: "encryptionkey@0"
I20260620 00:07:05.763387 5812 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:07:05.764103 5826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:05.764290 5812 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:07:05.764395 5812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "1e935be8d4634cb5aade3c1f20ce3bd7"
format_stamp: "Formatted at 2026-06-20 00:07:05 on dist-test-slave-grqt"
server_key: "b807abff7bdd38359ec1674117a00ba7"
server_key_iv: "2caea1847ed4b2da6e4e65c0d12d70b7"
server_key_version: "encryptionkey@0"
I20260620 00:07:05.764469 5812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:05.774168 5812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:05.774466 5812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:05.774613 5812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:05.778362 5812 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35381
I20260620 00:07:05.778398 5878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35381 every 8 connection(s)
I20260620 00:07:05.778756 5812 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:07:05.779783 5879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:05.781602 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 5812
I20260620 00:07:05.781710 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:07:05.781893 392 external_mini_cluster.cc:1428] Setting key 922d81d551f7121fb4eb4d6b3d8a218d
I20260620 00:07:05.782230 5879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e935be8d4634cb5aade3c1f20ce3bd7: Bootstrap starting.
I20260620 00:07:05.782941 5879 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e935be8d4634cb5aade3c1f20ce3bd7: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:05.783258 5879 log.cc:824] T 00000000000000000000000000000000 P 1e935be8d4634cb5aade3c1f20ce3bd7: Log is configured to *not* fsync() on all Append() calls
I20260620 00:07:05.783950 5879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e935be8d4634cb5aade3c1f20ce3bd7: No bootstrap required, opened a new log
I20260620 00:07:05.785795 5879 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e935be8d4634cb5aade3c1f20ce3bd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e935be8d4634cb5aade3c1f20ce3bd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35381 } }
I20260620 00:07:05.785956 5879 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e935be8d4634cb5aade3c1f20ce3bd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:05.786002 5879 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e935be8d4634cb5aade3c1f20ce3bd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e935be8d4634cb5aade3c1f20ce3bd7, State: Initialized, Role: FOLLOWER
I20260620 00:07:05.786144 5879 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e935be8d4634cb5aade3c1f20ce3bd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e935be8d4634cb5aade3c1f20ce3bd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35381 } }
I20260620 00:07:05.786226 5879 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e935be8d4634cb5aade3c1f20ce3bd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:07:05.786263 5879 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e935be8d4634cb5aade3c1f20ce3bd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:07:05.786309 5879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e935be8d4634cb5aade3c1f20ce3bd7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:05.787030 5879 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e935be8d4634cb5aade3c1f20ce3bd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e935be8d4634cb5aade3c1f20ce3bd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35381 } }
I20260620 00:07:05.787153 5879 leader_election.cc:302] T 00000000000000000000000000000000 P 1e935be8d4634cb5aade3c1f20ce3bd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e935be8d4634cb5aade3c1f20ce3bd7; no voters:
I20260620 00:07:05.787348 5879 leader_election.cc:288] T 00000000000000000000000000000000 P 1e935be8d4634cb5aade3c1f20ce3bd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:07:05.787386 5884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e935be8d4634cb5aade3c1f20ce3bd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:07:05.787529 5884 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e935be8d4634cb5aade3c1f20ce3bd7 [term 1 LEADER]: Becoming Leader. State: Replica: 1e935be8d4634cb5aade3c1f20ce3bd7, State: Running, Role: LEADER
I20260620 00:07:05.787608 5879 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e935be8d4634cb5aade3c1f20ce3bd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:07:05.787619 5884 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e935be8d4634cb5aade3c1f20ce3bd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e935be8d4634cb5aade3c1f20ce3bd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35381 } }
I20260620 00:07:05.787870 5882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:05.782490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41134 (local address 127.0.98.62:35381)
0620 00:07:05.782708 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:05.782716 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:05.782750 (+ 34us) server_negotiation.cc:407] Connection header received
0620 00:07:05.783391 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:05.783401 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:05.783485 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:05.783661 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:05.783855 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:05.784810 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:05.785639 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:05.786131 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:05.786254 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:05.786954 (+ 700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:05.786957 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:05.786958 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:07:05.787130 (+ 172us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:05.787191 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:07:05.787204 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:05.787281 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:05.787571 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 00:07:05.787653 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":67,"threads_started":1}
I20260620 00:07:05.787950 5884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e935be8d4634cb5aade3c1f20ce3bd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e935be8d4634cb5aade3c1f20ce3bd7. Latest consensus state: current_term: 1 leader_uuid: "1e935be8d4634cb5aade3c1f20ce3bd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e935be8d4634cb5aade3c1f20ce3bd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35381 } } }
I20260620 00:07:05.788020 5884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e935be8d4634cb5aade3c1f20ce3bd7 [sys.catalog]: This master's current role is: LEADER
I20260620 00:07:05.788242 5889 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:07:05.788712 5889 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:07:05.789140 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:07:05.789294 5885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e935be8d4634cb5aade3c1f20ce3bd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e935be8d4634cb5aade3c1f20ce3bd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e935be8d4634cb5aade3c1f20ce3bd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35381 } } }
I20260620 00:07:05.790091 5885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e935be8d4634cb5aade3c1f20ce3bd7 [sys.catalog]: This master's current role is: LEADER
I20260620 00:07:05.794533 5889 catalog_manager.cc:1347] Generated new cluster ID: 3558dc48de6647518247fba8b2ddca1b
I20260620 00:07:05.794605 5889 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:07:05.809113 5889 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:07:05.809934 5889 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:07:05.819244 5889 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e935be8d4634cb5aade3c1f20ce3bd7: Generated new TSK 0
I20260620 00:07:05.819464 5889 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:07:05.860915 5882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:05.825723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41138 (local address 127.0.98.62:35381)
0620 00:07:05.825776 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:05.825778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:05.825874 (+ 96us) server_negotiation.cc:407] Connection header received
0620 00:07:05.825938 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:05.825939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:05.825952 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:05.826023 (+ 71us) server_negotiation.cc:226] Negotiated authn=JWT
0620 00:07:05.826131 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:05.826919 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:05.827749 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:05.828092 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:05.828163 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:05.828364 (+ 201us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 00:07:05.860614 (+ 32250us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 00:07:05.860832 (+ 218us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41138: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":24}
W20260620 00:07:05.861001 5882 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41138: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 00:07:05.861039 5908 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:05.825651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35381 (local address 127.0.0.1:41138)
0620 00:07:05.825803 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:05.825813 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:05.825868 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:05.826025 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:05.826026 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:05.826027 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0620 00:07:05.826074 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:05.826075 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:05.827035 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:05.827036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:05.827662 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:05.827665 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:05.828196 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:05.828201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:05.828253 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:05.828258 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 00:07:05.860803 (+ 32545us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 00:07:05.860924 (+ 121us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:35381: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
I20260620 00:07:05.861490 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 5812
2026-06-20T00:07:05Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (219 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (92 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (89 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:07:06 dist-test-slave-grqt krb5kdc[5920](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:07:06 dist-test-slave-grqt krb5kdc[5920](info): set up 2 sockets
Jun 20 00:07:06 dist-test-slave-grqt krb5kdc[5920](info): commencing operation
krb5kdc: starting...
W20260620 00:07:07.109647 392 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:07:07 dist-test-slave-grqt krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:07:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:07:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:07:07.237829 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46667
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46667
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:07:07.319954 5933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:07.320240 5933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:07.320277 5933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:07.321521 5933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:07:07.321592 5933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:07.321616 5933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:07.321637 5933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:07:07.321666 5933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:07:07.323338 5933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:07.323609 5933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:07.323846 5933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:07.328872 5939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:07.328931 5941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:07.328897 5938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:07.328945 5933 server_base.cc:1034] running on GCE node
I20260620 00:07:07.329332 5933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:07.329686 5933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:07.330874 5933 hybrid_clock.cc:648] HybridClock initialized: now 1781914027330860 us; error 44 us; skew 500 ppm
Jun 20 00:07:07 dist-test-slave-grqt krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:07.333483 5933 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:07:07.333979 5933 webserver.cc:460] Webserver started at http://127.0.98.62:38839/ using document root <none> and password file <none>
I20260620 00:07:07.334229 5933 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:07.334299 5933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:07.334447 5933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:07.335546 5933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "be3294e2b11e41d9b8e259ebf69053b1"
format_stamp: "Formatted at 2026-06-20 00:07:07 on dist-test-slave-grqt"
server_key: "2b57223ecc7b453f8f21381a3ec4b147"
server_key_iv: "2865198a0f98341a97b7ba1810651afd"
server_key_version: "encryptionkey@0"
I20260620 00:07:07.335920 5933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "be3294e2b11e41d9b8e259ebf69053b1"
format_stamp: "Formatted at 2026-06-20 00:07:07 on dist-test-slave-grqt"
server_key: "2b57223ecc7b453f8f21381a3ec4b147"
server_key_iv: "2865198a0f98341a97b7ba1810651afd"
server_key_version: "encryptionkey@0"
I20260620 00:07:07.337339 5933 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:07:07.338215 5948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:07.338480 5933 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:07.338600 5933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "be3294e2b11e41d9b8e259ebf69053b1"
format_stamp: "Formatted at 2026-06-20 00:07:07 on dist-test-slave-grqt"
server_key: "2b57223ecc7b453f8f21381a3ec4b147"
server_key_iv: "2865198a0f98341a97b7ba1810651afd"
server_key_version: "encryptionkey@0"
I20260620 00:07:07.338689 5933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:07.347657 5933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:07.349838 5933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:07.350008 5933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:07.354117 5933 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46667
I20260620 00:07:07.354153 6000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46667 every 8 connection(s)
I20260620 00:07:07.354671 5933 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:07:07.355309 6001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:07.357774 6001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be3294e2b11e41d9b8e259ebf69053b1: Bootstrap starting.
I20260620 00:07:07.358342 6001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be3294e2b11e41d9b8e259ebf69053b1: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:07.358628 6001 log.cc:824] T 00000000000000000000000000000000 P be3294e2b11e41d9b8e259ebf69053b1: Log is configured to *not* fsync() on all Append() calls
I20260620 00:07:07.359270 6001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be3294e2b11e41d9b8e259ebf69053b1: No bootstrap required, opened a new log
I20260620 00:07:07.360810 6001 raft_consensus.cc:348] T 00000000000000000000000000000000 P be3294e2b11e41d9b8e259ebf69053b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3294e2b11e41d9b8e259ebf69053b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46667 } }
I20260620 00:07:07.360942 6001 raft_consensus.cc:374] T 00000000000000000000000000000000 P be3294e2b11e41d9b8e259ebf69053b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:07.360987 6001 raft_consensus.cc:729] T 00000000000000000000000000000000 P be3294e2b11e41d9b8e259ebf69053b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be3294e2b11e41d9b8e259ebf69053b1, State: Initialized, Role: FOLLOWER
I20260620 00:07:07.361146 6001 consensus_queue.cc:260] T 00000000000000000000000000000000 P be3294e2b11e41d9b8e259ebf69053b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3294e2b11e41d9b8e259ebf69053b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46667 } }
I20260620 00:07:07.361227 6001 raft_consensus.cc:388] T 00000000000000000000000000000000 P be3294e2b11e41d9b8e259ebf69053b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:07:07.361266 6001 raft_consensus.cc:482] T 00000000000000000000000000000000 P be3294e2b11e41d9b8e259ebf69053b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:07:07.361313 6001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be3294e2b11e41d9b8e259ebf69053b1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:07.362255 6001 raft_consensus.cc:504] T 00000000000000000000000000000000 P be3294e2b11e41d9b8e259ebf69053b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3294e2b11e41d9b8e259ebf69053b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46667 } }
I20260620 00:07:07.362341 6001 leader_election.cc:302] T 00000000000000000000000000000000 P be3294e2b11e41d9b8e259ebf69053b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be3294e2b11e41d9b8e259ebf69053b1; no voters:
I20260620 00:07:07.362511 6001 leader_election.cc:288] T 00000000000000000000000000000000 P be3294e2b11e41d9b8e259ebf69053b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:07:07.362579 6004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be3294e2b11e41d9b8e259ebf69053b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:07:07.362762 6004 raft_consensus.cc:686] T 00000000000000000000000000000000 P be3294e2b11e41d9b8e259ebf69053b1 [term 1 LEADER]: Becoming Leader. State: Replica: be3294e2b11e41d9b8e259ebf69053b1, State: Running, Role: LEADER
I20260620 00:07:07.362769 6001 sys_catalog.cc:564] T 00000000000000000000000000000000 P be3294e2b11e41d9b8e259ebf69053b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:07:07.362849 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 5933
I20260620 00:07:07.362936 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:07:07.362924 6004 consensus_queue.cc:237] T 00000000000000000000000000000000 P be3294e2b11e41d9b8e259ebf69053b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3294e2b11e41d9b8e259ebf69053b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46667 } }
I20260620 00:07:07.363148 392 external_mini_cluster.cc:1428] Setting key 017d0814e6516f15a50b123014ee9b6d
I20260620 00:07:07.363270 6005 sys_catalog.cc:455] T 00000000000000000000000000000000 P be3294e2b11e41d9b8e259ebf69053b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be3294e2b11e41d9b8e259ebf69053b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3294e2b11e41d9b8e259ebf69053b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46667 } } }
I20260620 00:07:07.363399 6005 sys_catalog.cc:458] T 00000000000000000000000000000000 P be3294e2b11e41d9b8e259ebf69053b1 [sys.catalog]: This master's current role is: LEADER
I20260620 00:07:07.363857 6013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:07:07.364169 6006 sys_catalog.cc:455] T 00000000000000000000000000000000 P be3294e2b11e41d9b8e259ebf69053b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be3294e2b11e41d9b8e259ebf69053b1. Latest consensus state: current_term: 1 leader_uuid: "be3294e2b11e41d9b8e259ebf69053b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3294e2b11e41d9b8e259ebf69053b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46667 } } }
I20260620 00:07:07.364286 6006 sys_catalog.cc:458] T 00000000000000000000000000000000 P be3294e2b11e41d9b8e259ebf69053b1 [sys.catalog]: This master's current role is: LEADER
I20260620 00:07:07.364444 6013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:07:07.367189 6013 catalog_manager.cc:1347] Generated new cluster ID: dbff722635f14f919e9af039715edc6e
I20260620 00:07:07.367244 6013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 00:07:07 dist-test-slave-grqt krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:07:07.374195 6017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:07.363821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58358 (local address 127.0.98.62:46667)
0620 00:07:07.364027 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:07.364037 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:07.364517 (+ 480us) server_negotiation.cc:407] Connection header received
0620 00:07:07.364731 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:07.364740 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:07.364798 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:07.364896 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:07.365650 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:07.367040 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:07.367969 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:07.368405 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:07.368542 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:07.370898 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:07.370901 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:07.370901 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:07.370911 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:07.372590 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:07.372971 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:07.372974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:07.372975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:07.373070 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:07.373280 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:07.373281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:07.373281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:07.373584 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:07.373768 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:07.373838 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 00:07:07.373976 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":68,"threads_started":1}
I20260620 00:07:07.377489 6013 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:07:07.378198 6013 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:07:07.383844 6013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be3294e2b11e41d9b8e259ebf69053b1: Generated new TSK 0
I20260620 00:07:07.384024 6013 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:07:07.411774 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46667
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:07:07.493903 6027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:07.494128 6027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:07.494154 6027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:07.495361 6027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:07.495412 6027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:07.495440 6027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:07:07.497040 6027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:07.497298 6027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:07.497582 6027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:07.502992 6033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:07.502962 6035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:07.502969 6032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:07.503019 6027 server_base.cc:1034] running on GCE node
I20260620 00:07:07.503438 6027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:07.503734 6027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:07.504911 6027 hybrid_clock.cc:648] HybridClock initialized: now 1781914027504901 us; error 49 us; skew 500 ppm
Jun 20 00:07:07 dist-test-slave-grqt krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:07.507603 6027 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:07:07.508050 6027 webserver.cc:460] Webserver started at http://127.0.98.1:41501/ using document root <none> and password file <none>
I20260620 00:07:07.508352 6027 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:07.508423 6027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:07.508577 6027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:07.509727 6027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-0/data/instance:
uuid: "40091a59d90d4d0ba911deb1d7dd2f58"
format_stamp: "Formatted at 2026-06-20 00:07:07 on dist-test-slave-grqt"
server_key: "7ae7158b16014e44523f7f7e8e9d5642"
server_key_iv: "88126dbe59eb199987b722c1d63ac362"
server_key_version: "encryptionkey@0"
I20260620 00:07:07.510085 6027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-0/wal/instance:
uuid: "40091a59d90d4d0ba911deb1d7dd2f58"
format_stamp: "Formatted at 2026-06-20 00:07:07 on dist-test-slave-grqt"
server_key: "7ae7158b16014e44523f7f7e8e9d5642"
server_key_iv: "88126dbe59eb199987b722c1d63ac362"
server_key_version: "encryptionkey@0"
I20260620 00:07:07.511443 6027 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:07:07.512275 6042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:07.512481 6027 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:07.512579 6027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-0/wal
uuid: "40091a59d90d4d0ba911deb1d7dd2f58"
format_stamp: "Formatted at 2026-06-20 00:07:07 on dist-test-slave-grqt"
server_key: "7ae7158b16014e44523f7f7e8e9d5642"
server_key_iv: "88126dbe59eb199987b722c1d63ac362"
server_key_version: "encryptionkey@0"
I20260620 00:07:07.512677 6027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:07.531795 6027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:07.533957 6027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:07.534124 6027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:07.534382 6027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:07:07.534716 6027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:07:07.534771 6027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:07.534816 6027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:07:07.534842 6027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:07.541128 6027 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35617
I20260620 00:07:07.541285 6155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35617 every 8 connection(s)
I20260620 00:07:07.541625 6027 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:07:07.546739 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 6027
I20260620 00:07:07.546837 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:07:07.547132 392 external_mini_cluster.cc:1428] Setting key 50cd3fa13c2b646e78155554a4b77c68
Jun 20 00:07:07 dist-test-slave-grqt krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:07:07.551817 6017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:07.542433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44513 (local address 127.0.98.62:46667)
0620 00:07:07.542509 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:07.542512 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:07.542715 (+ 203us) server_negotiation.cc:407] Connection header received
0620 00:07:07.543663 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:07.543664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:07.543680 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:07.543739 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:07.544332 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:07.544833 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:07.545963 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:07.546174 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:07.546245 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:07.548865 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:07.548867 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:07.548867 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:07.548875 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:07.550373 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:07.550828 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:07.550829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:07.550830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:07.550891 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:07.551147 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:07.551148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:07.551149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:07.551269 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:07.551347 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:07.551631 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 00:07:07.551721 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 00:07:07.552034 6158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:07.542362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46667 (local address 127.0.98.1:44513)
0620 00:07:07.542540 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:07.542578 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:07.543554 (+ 976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:07.543795 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:07.543799 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:07.544059 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:07.544252 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:07.544256 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:07.544933 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:07.544934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:07.545874 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:07.545877 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:07.546276 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:07.546277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:07.546398 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:07.547075 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:07.547086 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:07.548755 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:07.550480 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:07.550488 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:07.550491 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:07.550733 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:07.550958 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:07.550959 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:07.550959 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:07.551075 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:07.551338 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:07.551345 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:07.551494 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 00:07:07.551608 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 00:07:07.551786 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":71,"threads_started":1}
I20260620 00:07:07.552587 6156 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46667
I20260620 00:07:07.552709 6156 heartbeater.cc:464] Registering TS with master...
I20260620 00:07:07.552893 6156 heartbeater.cc:511] Master 127.0.98.62:46667 requested a full tablet report, sending...
I20260620 00:07:07.553578 5965 ts_manager.cc:194] Registered new tserver with Master: 40091a59d90d4d0ba911deb1d7dd2f58 (127.0.98.1:35617)
I20260620 00:07:07.554556 5965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44513
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:07:07.577103 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46667
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:07:07.665501 6161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:07.665737 6161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:07.665762 6161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:07.666987 6161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:07.667032 6161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:07.667060 6161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:07:07.668620 6161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:07.668886 6161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:07.669133 6161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:07.674271 6169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:07.674283 6166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:07.674338 6167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:07.674636 6161 server_base.cc:1034] running on GCE node
I20260620 00:07:07.674922 6161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:07.675195 6161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:07.676430 6161 hybrid_clock.cc:648] HybridClock initialized: now 1781914027676416 us; error 32 us; skew 500 ppm
Jun 20 00:07:07 dist-test-slave-grqt krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:07.679144 6161 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:07:07.679636 6161 webserver.cc:460] Webserver started at http://127.0.98.2:38163/ using document root <none> and password file <none>
I20260620 00:07:07.679904 6161 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:07.679977 6161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:07.680135 6161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:07.681234 6161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-1/data/instance:
uuid: "e09d873c4a0a46f69e3dabf93b9f04f9"
format_stamp: "Formatted at 2026-06-20 00:07:07 on dist-test-slave-grqt"
server_key: "697f3e15fb98c07e8970e820e689a2c2"
server_key_iv: "50aeb44309a6880383b2e8f1ce80e3b4"
server_key_version: "encryptionkey@0"
I20260620 00:07:07.681675 6161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e09d873c4a0a46f69e3dabf93b9f04f9"
format_stamp: "Formatted at 2026-06-20 00:07:07 on dist-test-slave-grqt"
server_key: "697f3e15fb98c07e8970e820e689a2c2"
server_key_iv: "50aeb44309a6880383b2e8f1ce80e3b4"
server_key_version: "encryptionkey@0"
I20260620 00:07:07.683065 6161 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:07.684043 6176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:07.684290 6161 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:07.684403 6161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-1/wal
uuid: "e09d873c4a0a46f69e3dabf93b9f04f9"
format_stamp: "Formatted at 2026-06-20 00:07:07 on dist-test-slave-grqt"
server_key: "697f3e15fb98c07e8970e820e689a2c2"
server_key_iv: "50aeb44309a6880383b2e8f1ce80e3b4"
server_key_version: "encryptionkey@0"
I20260620 00:07:07.684484 6161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:07.700485 6161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:07.702811 6161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:07.702930 6161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:07.703111 6161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:07:07.703445 6161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:07:07.703481 6161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:07.703529 6161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:07:07.703555 6161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:07.709466 6161 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45389
I20260620 00:07:07.709544 6289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45389 every 8 connection(s)
I20260620 00:07:07.709882 6161 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:07:07.712391 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 6161
I20260620 00:07:07.712487 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:07:07.712704 392 external_mini_cluster.cc:1428] Setting key 4355143fd1b2ea54a35ac20acca388e8
Jun 20 00:07:07 dist-test-slave-grqt krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 00:07:07.721004 6017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:07.710713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39925 (local address 127.0.98.62:46667)
0620 00:07:07.710783 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:07.710786 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:07.711020 (+ 234us) server_negotiation.cc:407] Connection header received
0620 00:07:07.712007 (+ 987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:07.712009 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:07.712020 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:07.712079 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:07.712827 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:07.713600 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:07.714864 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:07.715053 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:07.715114 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:07.717808 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:07.717810 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:07.717810 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:07.717819 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:07.719444 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:07.719945 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:07.719947 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:07.719947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:07.720015 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:07.720296 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:07.720297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:07.720297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:07.720431 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:07.720511 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:07.720806 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 00:07:07.720903 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 00:07:07.721213 6292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:07.710705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46667 (local address 127.0.98.2:39925)
0620 00:07:07.710889 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:07.710928 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:07.711889 (+ 961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:07.712138 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:07.712142 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:07.712438 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:07.712703 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:07.712709 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:07.713733 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:07.713734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:07.714752 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:07.714755 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:07.715146 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:07.715147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:07.715354 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:07.716049 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:07.716058 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:07.717695 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:07.719571 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:07.719580 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:07.719582 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:07.719847 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:07.720100 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:07.720101 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:07.720101 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:07.720226 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:07.720501 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:07.720507 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:07.720668 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 00:07:07.720783 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 00:07:07.720969 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":102,"threads_started":1}
I20260620 00:07:07.721868 6290 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46667
I20260620 00:07:07.721998 6290 heartbeater.cc:464] Registering TS with master...
I20260620 00:07:07.722189 6290 heartbeater.cc:511] Master 127.0.98.62:46667 requested a full tablet report, sending...
I20260620 00:07:07.722577 5965 ts_manager.cc:194] Registered new tserver with Master: e09d873c4a0a46f69e3dabf93b9f04f9 (127.0.98.2:45389)
I20260620 00:07:07.722976 5965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39925
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:07:07.740149 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46667
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:07:07.828233 6295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:07.828457 6295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:07.828482 6295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:07.829738 6295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:07.829787 6295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:07.829815 6295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:07:07.831384 6295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:07.831656 6295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:07.831903 6295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:07.837199 6300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:07.837206 6301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:07.837306 6295 server_base.cc:1034] running on GCE node
W20260620 00:07:07.837206 6303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:07.837654 6295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:07.837878 6295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:07.839040 6295 hybrid_clock.cc:648] HybridClock initialized: now 1781914027839025 us; error 32 us; skew 500 ppm
Jun 20 00:07:07 dist-test-slave-grqt krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:07.841961 6295 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:07:07.842514 6295 webserver.cc:460] Webserver started at http://127.0.98.3:41215/ using document root <none> and password file <none>
I20260620 00:07:07.842790 6295 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:07.842864 6295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:07.843026 6295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:07.844089 6295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-2/data/instance:
uuid: "22da3dbdcbe04d45a500ca1b80061828"
format_stamp: "Formatted at 2026-06-20 00:07:07 on dist-test-slave-grqt"
server_key: "bc909f50ef56e1c7b2827dc55b8ab2f6"
server_key_iv: "7d9511cf685521a26635e40a53390083"
server_key_version: "encryptionkey@0"
I20260620 00:07:07.844491 6295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-2/wal/instance:
uuid: "22da3dbdcbe04d45a500ca1b80061828"
format_stamp: "Formatted at 2026-06-20 00:07:07 on dist-test-slave-grqt"
server_key: "bc909f50ef56e1c7b2827dc55b8ab2f6"
server_key_iv: "7d9511cf685521a26635e40a53390083"
server_key_version: "encryptionkey@0"
I20260620 00:07:07.845927 6295 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:07:07.846745 6310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:07.847004 6295 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:07:07.847090 6295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-2/wal
uuid: "22da3dbdcbe04d45a500ca1b80061828"
format_stamp: "Formatted at 2026-06-20 00:07:07 on dist-test-slave-grqt"
server_key: "bc909f50ef56e1c7b2827dc55b8ab2f6"
server_key_iv: "7d9511cf685521a26635e40a53390083"
server_key_version: "encryptionkey@0"
I20260620 00:07:07.847186 6295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:07.862895 6295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:07.865590 6295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:07.865758 6295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:07.866003 6295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:07:07.866317 6295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:07:07.866351 6295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:07.866395 6295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:07:07.866422 6295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:07.872494 6295 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35149
I20260620 00:07:07.872566 6423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35149 every 8 connection(s)
I20260620 00:07:07.872884 6295 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:07:07.875741 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 6295
I20260620 00:07:07.875836 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:07:07.875988 392 external_mini_cluster.cc:1428] Setting key 96bab57ac57ccbed98a857ef71a098dc
Jun 20 00:07:07 dist-test-slave-grqt krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:07:07.882834 6017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:07.873694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47963 (local address 127.0.98.62:46667)
0620 00:07:07.873789 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:07.873791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:07.873955 (+ 164us) server_negotiation.cc:407] Connection header received
0620 00:07:07.874896 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:07.874897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:07.874909 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:07.874973 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:07.875590 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:07.876319 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:07.877539 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:07.877728 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:07.877788 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:07.879816 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:07.879818 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:07.879818 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:07.879824 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:07.881307 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:07.881840 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:07.881841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:07.881842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:07.881899 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:07.882209 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:07.882209 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:07.882210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:07.882320 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:07.882372 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:07.882695 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 00:07:07.882766 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 00:07:07.883040 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:07.873670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46667 (local address 127.0.98.3:47963)
0620 00:07:07.873827 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:07.873870 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:07.874797 (+ 927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:07.875030 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:07.875034 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:07.875296 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:07.875488 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:07.875492 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:07.876421 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:07.876422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:07.877465 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:07.877471 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:07.877799 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:07.877800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:07.877894 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:07.878360 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:07.878369 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:07.879704 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:07.881455 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:07.881467 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:07.881469 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:07.881731 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:07.881989 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:07.881990 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:07.881990 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:07.882104 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:07.882404 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:07.882409 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:07.882535 (+ 126us) client_negotiation.cc:769] Sending connection context
0620 00:07:07.882653 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 00:07:07.882785 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
I20260620 00:07:07.883564 6424 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46667
I20260620 00:07:07.883656 6424 heartbeater.cc:464] Registering TS with master...
I20260620 00:07:07.883845 6424 heartbeater.cc:511] Master 127.0.98.62:46667 requested a full tablet report, sending...
I20260620 00:07:07.884301 5965 ts_manager.cc:194] Registered new tserver with Master: 22da3dbdcbe04d45a500ca1b80061828 (127.0.98.3:35149)
I20260620 00:07:07.884694 5965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47963
I20260620 00:07:07.888146 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/krb5kdc/test-admin.keytab.
Jun 20 00:07:07 dist-test-slave-grqt krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:07.902570 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 00:07:07.904500 6431 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:07.903582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55032 (local address 127.0.98.1:35617)
0620 00:07:07.903765 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:07.903770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:07.903780 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:07:07.903806 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:07.903807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:07.903822 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:07.903892 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:07.904404 (+ 512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55032: BlockingRecv error: recv got EOF from 127.0.0.1:55032 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":105,"threads_started":1}
W20260620 00:07:07.904735 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/krb5kdc/krb5cc' of type FILE
Jun 20 00:07:07 dist-test-slave-grqt krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:07.906015 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 00:07:07 dist-test-slave-grqt krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:07.907094 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 00:07:07 dist-test-slave-grqt krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 00:07:07.916365 6431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:07.907772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55044 (local address 127.0.98.1:35617)
0620 00:07:07.907868 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:07.907870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:07.907880 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:07:07.907909 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:07.907909 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:07.907922 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:07.907984 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:07.908445 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:07.909305 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:07.910109 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:07.910446 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:07.910510 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:07.912791 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:07.912839 (+ 48us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:07.912839 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:07.912846 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:07.914918 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:07.915375 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:07.915379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:07.915379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:07.915448 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:07.915658 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:07.915659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:07.915660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:07.915832 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:07.916024 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:07.916077 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 00:07:07.916159 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 00:07:07.918649 6431 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:07.918039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55052 (local address 127.0.98.1:35617)
0620 00:07:07.918093 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:07.918095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:07.918103 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:07:07.918163 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:07.918164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:07.918176 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:07.918275 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:07.918596 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55052: BlockingRecv error: recv got EOF from 127.0.0.1:55052 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260620 00:07:07.918845 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/krb5kdc/krb5cc' of type FILE
Jun 20 00:07:07 dist-test-slave-grqt krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:07.919919 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 00:07:07 dist-test-slave-grqt krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:07.920804 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 00:07:07 dist-test-slave-grqt krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 00:07:07.928633 6431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:07.921445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55066 (local address 127.0.98.1:35617)
0620 00:07:07.921501 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:07.921503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:07.921556 (+ 53us) server_negotiation.cc:407] Connection header received
0620 00:07:07.921634 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:07.921635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:07.921650 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:07.921721 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:07.922133 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:07.922651 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:07.923359 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:07.923573 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:07.923664 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:07.925979 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:07.925981 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:07.925981 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:07.925988 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:07.927398 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:07.927913 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:07.927913 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:07.927914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:07.927967 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:07.928163 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:07.928163 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:07.928164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:07.928280 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:07.928356 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:07.928497 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 00:07:07.928566 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260620 00:07:07.930574 6431 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:07.929763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55072 (local address 127.0.98.1:35617)
0620 00:07:07.929800 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:07.929802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:07.929867 (+ 65us) server_negotiation.cc:407] Connection header received
0620 00:07:07.929911 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:07.929911 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:07.929919 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:07.929968 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:07.930308 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55072: BlockingRecv error: recv got EOF from 127.0.0.1:55072 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 00:07:07.930766 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914001714546-392-0/krb5kdc/krb5cc' of type FILE
Jun 20 00:07:07 dist-test-slave-grqt krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:07.931816 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 00:07:07 dist-test-slave-grqt krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:07.932749 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 00:07:07 dist-test-slave-grqt krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 00:07:07.940239 6431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:07.933351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55084 (local address 127.0.98.1:35617)
0620 00:07:07.933454 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:07.933456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:07.933541 (+ 85us) server_negotiation.cc:407] Connection header received
0620 00:07:07.933615 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:07.933616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:07.933626 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:07.933678 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:07.934061 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:07.934546 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:07.935279 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:07.935458 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:07.935527 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:07.937672 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:07.937673 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:07.937674 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:07.937682 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:07.939049 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:07.939469 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:07.939470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:07.939471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:07.939529 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:07.939724 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:07.939725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:07.939726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:07.939831 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:07.939934 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:07.940036 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 00:07:07.940133 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 00:07:07.940938 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 6027
I20260620 00:07:07.946105 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 6161
I20260620 00:07:07.950629 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 6295
I20260620 00:07:07.954806 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 5933
2026-06-20T00:07:07Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1913 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:07:08 dist-test-slave-grqt krb5kdc[6444](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:07:08 dist-test-slave-grqt krb5kdc[6444](info): set up 2 sockets
Jun 20 00:07:08 dist-test-slave-grqt krb5kdc[6444](info): commencing operation
krb5kdc: starting...
W20260620 00:07:09.029639 392 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:07:09 dist-test-slave-grqt krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:07:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:07:09Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260620 00:07:09.160760 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45835
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:07:09.253048 6457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:09.253283 6457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:09.253306 6457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:09.254531 6457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:07:09.254582 6457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:09.254597 6457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:09.254611 6457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:07:09.254622 6457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:07:09.256255 6457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:09.256546 6457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:09.256822 6457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:09.262048 6465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:09.262058 6462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:09.262042 6463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:09.262475 6457 server_base.cc:1034] running on GCE node
I20260620 00:07:09.262759 6457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:09.263077 6457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:09.264274 6457 hybrid_clock.cc:648] HybridClock initialized: now 1781914029264259 us; error 33 us; skew 500 ppm
Jun 20 00:07:09 dist-test-slave-grqt krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914029, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:09.267073 6457 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260620 00:07:09.267561 6457 webserver.cc:460] Webserver started at http://127.0.98.62:46881/ using document root <none> and password file <none>
I20260620 00:07:09.267810 6457 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:09.267858 6457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:09.267992 6457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:09.269100 6457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1"
format_stamp: "Formatted at 2026-06-20 00:07:09 on dist-test-slave-grqt"
server_key: "5770e53022a5c4674defdd81895e7ba0"
server_key_iv: "31e5673fc5dcfc20d83435fc62743dd2"
server_key_version: "encryptionkey@0"
I20260620 00:07:09.269503 6457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1"
format_stamp: "Formatted at 2026-06-20 00:07:09 on dist-test-slave-grqt"
server_key: "5770e53022a5c4674defdd81895e7ba0"
server_key_iv: "31e5673fc5dcfc20d83435fc62743dd2"
server_key_version: "encryptionkey@0"
I20260620 00:07:09.270967 6457 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:09.271808 6472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:09.272008 6457 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:09.272099 6457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1"
format_stamp: "Formatted at 2026-06-20 00:07:09 on dist-test-slave-grqt"
server_key: "5770e53022a5c4674defdd81895e7ba0"
server_key_iv: "31e5673fc5dcfc20d83435fc62743dd2"
server_key_version: "encryptionkey@0"
I20260620 00:07:09.272156 6457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:09.280154 6457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:09.282487 6457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:09.282608 6457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:09.286558 6457 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45835
I20260620 00:07:09.286639 6524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45835 every 8 connection(s)
I20260620 00:07:09.286962 6457 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:07:09.287976 6525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:09.290367 6525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1: Bootstrap starting.
I20260620 00:07:09.290992 6525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:09.291287 6525 log.cc:824] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1: Log is configured to *not* fsync() on all Append() calls
I20260620 00:07:09.292023 6525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1: No bootstrap required, opened a new log
I20260620 00:07:09.293663 6525 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } }
I20260620 00:07:09.293844 6525 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:09.293891 6525 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60dfbd8a9e3a4e4aaad18e7dac6d0ea1, State: Initialized, Role: FOLLOWER
I20260620 00:07:09.294087 6525 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } }
I20260620 00:07:09.294191 6525 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:07:09.294237 6525 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:07:09.294289 6525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:09.295135 6525 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } }
I20260620 00:07:09.295262 6525 leader_election.cc:302] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60dfbd8a9e3a4e4aaad18e7dac6d0ea1; no voters:
I20260620 00:07:09.295462 6525 leader_election.cc:288] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:07:09.295594 6528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:07:09.295733 6525 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:07:09.295748 6528 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [term 1 LEADER]: Becoming Leader. State: Replica: 60dfbd8a9e3a4e4aaad18e7dac6d0ea1, State: Running, Role: LEADER
I20260620 00:07:09.295938 6528 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } }
I20260620 00:07:09.296379 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 6457
I20260620 00:07:09.296546 6530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60dfbd8a9e3a4e4aaad18e7dac6d0ea1. Latest consensus state: current_term: 1 leader_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } }
I20260620 00:07:09.296615 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:07:09.296639 6530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [sys.catalog]: This master's current role is: LEADER
I20260620 00:07:09.296859 6529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } }
I20260620 00:07:09.296893 392 external_mini_cluster.cc:1428] Setting key 7d5acf1a088fee4d67c5f7aba374518a
I20260620 00:07:09.296918 6529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [sys.catalog]: This master's current role is: LEADER
I20260620 00:07:09.297770 6537 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:07:09.298409 6537 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:07:09.300290 6537 catalog_manager.cc:1347] Generated new cluster ID: 7bcc619fd37045da9bc9c98085719301
I20260620 00:07:09.300348 6537 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 00:07:09 dist-test-slave-grqt krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 00:07:09.309288 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:09.297751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39196 (local address 127.0.98.62:45835)
0620 00:07:09.297944 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:09.297955 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:09.297996 (+ 41us) server_negotiation.cc:407] Connection header received
0620 00:07:09.298139 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:09.298147 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:09.298265 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:09.298439 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:09.299123 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:09.300601 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:09.301527 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:09.301994 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:09.302151 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:09.304668 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:09.304673 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:09.304673 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:09.304687 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:09.307489 (+ 2802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:09.307919 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:09.307923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:09.307923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:09.308037 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:09.308261 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:09.308262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:09.308262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:09.308523 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:09.308709 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:09.308783 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 00:07:09.309052 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":120,"thread_start_us":75,"threads_started":1}
I20260620 00:07:09.309919 6537 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:07:09.310621 6537 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:07:09.312883 6537 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1: Generated new TSK 0
I20260620 00:07:09.313042 6537 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260620 00:07:09.349574 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45835
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:07:09.450371 6551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:09.450636 6551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:09.450665 6551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:09.452059 6551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:09.452118 6551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:09.452150 6551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:07:09.453848 6551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:09.454123 6551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:09.454392 6551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:09.459751 6557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:09.459753 6556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:09.459774 6559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:09.460058 6551 server_base.cc:1034] running on GCE node
I20260620 00:07:09.460279 6551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:09.460579 6551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:09.461747 6551 hybrid_clock.cc:648] HybridClock initialized: now 1781914029461726 us; error 33 us; skew 500 ppm
Jun 20 00:07:09 dist-test-slave-grqt krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914029, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:09.467885 6551 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 00:07:09.468416 6551 webserver.cc:460] Webserver started at http://127.0.98.1:40623/ using document root <none> and password file <none>
I20260620 00:07:09.468672 6551 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:09.468751 6551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:09.468919 6551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:09.470244 6551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-0/data/instance:
uuid: "ee53defd13044c57bbb138930c23dbf6"
format_stamp: "Formatted at 2026-06-20 00:07:09 on dist-test-slave-grqt"
server_key: "f6cc9eb0d7ad8123230204a56051cadb"
server_key_iv: "cd46cb4ad8bec409f36adf33a20e9c63"
server_key_version: "encryptionkey@0"
I20260620 00:07:09.470650 6551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ee53defd13044c57bbb138930c23dbf6"
format_stamp: "Formatted at 2026-06-20 00:07:09 on dist-test-slave-grqt"
server_key: "f6cc9eb0d7ad8123230204a56051cadb"
server_key_iv: "cd46cb4ad8bec409f36adf33a20e9c63"
server_key_version: "encryptionkey@0"
I20260620 00:07:09.472160 6551 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:07:09.472982 6566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:09.473222 6551 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:09.473336 6551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-0/wal
uuid: "ee53defd13044c57bbb138930c23dbf6"
format_stamp: "Formatted at 2026-06-20 00:07:09 on dist-test-slave-grqt"
server_key: "f6cc9eb0d7ad8123230204a56051cadb"
server_key_iv: "cd46cb4ad8bec409f36adf33a20e9c63"
server_key_version: "encryptionkey@0"
I20260620 00:07:09.473508 6551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:09.487931 6551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:09.490316 6551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:09.490513 6551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:09.490792 6551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:07:09.491146 6551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:07:09.491209 6551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:09.491253 6551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:07:09.491284 6551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:09.498481 6551 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44335
I20260620 00:07:09.498566 6679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44335 every 8 connection(s)
I20260620 00:07:09.498947 6551 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 00:07:09 dist-test-slave-grqt krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914029, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 00:07:09.505828 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 6551
I20260620 00:07:09.505972 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:07:09.506265 392 external_mini_cluster.cc:1428] Setting key dce6b49afd87ab0909282e8f4a7be0f1
I20260620 00:07:09.509750 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:09.500016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34835 (local address 127.0.98.62:45835)
0620 00:07:09.500113 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:09.500115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:09.500277 (+ 162us) server_negotiation.cc:407] Connection header received
0620 00:07:09.500799 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:09.500801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:09.500815 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:09.500875 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:09.501981 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:09.502511 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:09.503618 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:09.503830 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:09.503902 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:09.506622 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:09.506623 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:09.506624 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:09.506633 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:09.508221 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:09.508698 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:09.508700 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:09.508700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:09.508762 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:09.509021 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:09.509022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:09.509022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:09.509146 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:09.509230 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:09.509601 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 00:07:09.509680 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 00:07:09.510053 6682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:09.499844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45835 (local address 127.0.98.1:34835)
0620 00:07:09.500061 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:09.500162 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:09.500660 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:09.500932 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:09.500936 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:09.501600 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:09.501828 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:09.501834 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:09.502626 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:09.502627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:09.503514 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:09.503517 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:09.503959 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:09.503960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:09.504084 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:09.504689 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:09.504699 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:09.506475 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:09.508339 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:09.508348 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:09.508351 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:09.508603 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:09.508830 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:09.508831 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:09.508831 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:09.508949 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:09.509221 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:09.509228 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:09.509442 (+ 214us) client_negotiation.cc:769] Sending connection context
0620 00:07:09.509565 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 00:07:09.509756 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":86,"threads_started":1}
I20260620 00:07:09.510723 6680 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45835
I20260620 00:07:09.510958 6680 heartbeater.cc:464] Registering TS with master...
I20260620 00:07:09.511186 6680 heartbeater.cc:511] Master 127.0.98.62:45835 requested a full tablet report, sending...
I20260620 00:07:09.511816 6489 ts_manager.cc:194] Registered new tserver with Master: ee53defd13044c57bbb138930c23dbf6 (127.0.98.1:44335)
I20260620 00:07:09.512468 6489 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34835
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260620 00:07:09.539702 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45835
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:07:09.629271 6685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:09.629603 6685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:09.629632 6685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:09.630859 6685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:09.630908 6685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:09.630937 6685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:07:09.632477 6685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:09.632786 6685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:09.633051 6685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:09.638696 6690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:09.638686 6693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:09.638686 6691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:09.639086 6685 server_base.cc:1034] running on GCE node
I20260620 00:07:09.639336 6685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:09.639640 6685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:09.640807 6685 hybrid_clock.cc:648] HybridClock initialized: now 1781914029640773 us; error 53 us; skew 500 ppm
Jun 20 00:07:09 dist-test-slave-grqt krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914029, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:09.644026 6685 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 00:07:09.644562 6685 webserver.cc:460] Webserver started at http://127.0.98.2:42847/ using document root <none> and password file <none>
I20260620 00:07:09.644821 6685 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:09.644898 6685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:09.645066 6685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:09.646286 6685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-1/data/instance:
uuid: "4835482b92894a80985bac108abdac59"
format_stamp: "Formatted at 2026-06-20 00:07:09 on dist-test-slave-grqt"
server_key: "ca3b5c6986eb5adf7bef0f8ce2872390"
server_key_iv: "eedcda7d03fcc5be9d125b6a6fb84447"
server_key_version: "encryptionkey@0"
I20260620 00:07:09.646673 6685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4835482b92894a80985bac108abdac59"
format_stamp: "Formatted at 2026-06-20 00:07:09 on dist-test-slave-grqt"
server_key: "ca3b5c6986eb5adf7bef0f8ce2872390"
server_key_iv: "eedcda7d03fcc5be9d125b6a6fb84447"
server_key_version: "encryptionkey@0"
I20260620 00:07:09.648207 6685 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 00:07:09.649150 6700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:09.649533 6685 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:07:09.649654 6685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-1/wal
uuid: "4835482b92894a80985bac108abdac59"
format_stamp: "Formatted at 2026-06-20 00:07:09 on dist-test-slave-grqt"
server_key: "ca3b5c6986eb5adf7bef0f8ce2872390"
server_key_iv: "eedcda7d03fcc5be9d125b6a6fb84447"
server_key_version: "encryptionkey@0"
I20260620 00:07:09.649752 6685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:09.659636 6685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:09.662026 6685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:09.662212 6685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:09.662472 6685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:07:09.662810 6685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:07:09.662868 6685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:09.662909 6685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:07:09.662932 6685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:09.669049 6685 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34161
I20260620 00:07:09.669107 6813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34161 every 8 connection(s)
I20260620 00:07:09.669544 6685 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:07:09.675757 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 6685
I20260620 00:07:09.675900 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:07:09.676162 392 external_mini_cluster.cc:1428] Setting key e0117643acc170f551c525a6c8ad09ba
Jun 20 00:07:09 dist-test-slave-grqt krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914029, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 00:07:09.681011 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:09.670331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57255 (local address 127.0.98.62:45835)
0620 00:07:09.670404 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:09.670407 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:09.670628 (+ 221us) server_negotiation.cc:407] Connection header received
0620 00:07:09.671136 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:09.671137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:09.671151 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:09.671219 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:09.672172 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:09.672700 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:09.674010 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:09.674220 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:09.674290 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:09.677057 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:09.677058 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:09.677058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:09.677067 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:09.679112 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:09.679783 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:09.679784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:09.679785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:09.679847 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:09.680222 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:09.680224 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:09.680224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:09.680363 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:09.680456 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:09.680809 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 00:07:09.680907 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 00:07:09.681236 6816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:09.670296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45835 (local address 127.0.98.2:57255)
0620 00:07:09.670498 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:09.670536 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:09.671017 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:09.671307 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:09.671311 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:09.671869 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:09.672061 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:09.672066 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:09.672806 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:09.672807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:09.673901 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:09.673905 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:09.674330 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:09.674331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:09.674483 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:09.675104 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:09.675112 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:09.676926 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:09.679228 (+ 2302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:09.679237 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:09.679240 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:09.679664 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:09.679926 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:09.679927 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:09.679927 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:09.680086 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:09.680444 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:09.680451 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:09.680612 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 00:07:09.680774 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 00:07:09.680959 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":113,"threads_started":1}
I20260620 00:07:09.681941 6814 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45835
I20260620 00:07:09.682041 6814 heartbeater.cc:464] Registering TS with master...
I20260620 00:07:09.682315 6814 heartbeater.cc:511] Master 127.0.98.62:45835 requested a full tablet report, sending...
I20260620 00:07:09.682864 6489 ts_manager.cc:194] Registered new tserver with Master: 4835482b92894a80985bac108abdac59 (127.0.98.2:34161)
I20260620 00:07:09.683452 6489 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57255
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 00:07:09.705039 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45835
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:07:09.787981 6819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:09.788205 6819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:09.788230 6819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:09.789490 6819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:09.789562 6819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:09.789613 6819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:07:09.791075 6819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:09.791272 6819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:09.791457 6819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:09.796757 6825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:09.796775 6824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:09.796794 6827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:09.797060 6819 server_base.cc:1034] running on GCE node
I20260620 00:07:09.797427 6819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:09.797758 6819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:09.798961 6819 hybrid_clock.cc:648] HybridClock initialized: now 1781914029798942 us; error 30 us; skew 500 ppm
Jun 20 00:07:09 dist-test-slave-grqt krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914029, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:09.801651 6819 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 00:07:09.802147 6819 webserver.cc:460] Webserver started at http://127.0.98.3:41529/ using document root <none> and password file <none>
I20260620 00:07:09.802402 6819 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:09.802450 6819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:09.802585 6819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:09.803701 6819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-2/data/instance:
uuid: "87af16a18180461ca7afc0bff09802ab"
format_stamp: "Formatted at 2026-06-20 00:07:09 on dist-test-slave-grqt"
server_key: "f829a782da2388c379bbfad3fdfd23c3"
server_key_iv: "b0f73bea5106e65a66fcc4ec0c22820b"
server_key_version: "encryptionkey@0"
I20260620 00:07:09.804080 6819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-2/wal/instance:
uuid: "87af16a18180461ca7afc0bff09802ab"
format_stamp: "Formatted at 2026-06-20 00:07:09 on dist-test-slave-grqt"
server_key: "f829a782da2388c379bbfad3fdfd23c3"
server_key_iv: "b0f73bea5106e65a66fcc4ec0c22820b"
server_key_version: "encryptionkey@0"
I20260620 00:07:09.805627 6819 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:07:09.806452 6834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:09.806659 6819 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:09.806757 6819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-2/wal
uuid: "87af16a18180461ca7afc0bff09802ab"
format_stamp: "Formatted at 2026-06-20 00:07:09 on dist-test-slave-grqt"
server_key: "f829a782da2388c379bbfad3fdfd23c3"
server_key_iv: "b0f73bea5106e65a66fcc4ec0c22820b"
server_key_version: "encryptionkey@0"
I20260620 00:07:09.806816 6819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:09.832561 6819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:09.835116 6819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:09.835314 6819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:09.835558 6819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:07:09.835865 6819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:07:09.835927 6819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:09.835971 6819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:07:09.835999 6819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:09.842062 6819 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46179
I20260620 00:07:09.842132 6947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46179 every 8 connection(s)
I20260620 00:07:09.842492 6819 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 00:07:09 dist-test-slave-grqt krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914029, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 00:07:09.850760 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 6819
I20260620 00:07:09.850865 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:07:09.851053 392 external_mini_cluster.cc:1428] Setting key d2038da8f009a2e95391d0f9d7d709e9
I20260620 00:07:09.853082 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:09.843401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34633 (local address 127.0.98.62:45835)
0620 00:07:09.843455 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:09.843458 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:09.843597 (+ 139us) server_negotiation.cc:407] Connection header received
0620 00:07:09.844125 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:09.844126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:09.844138 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:09.844339 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:09.845446 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:09.845936 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:09.846944 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:09.847127 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:09.847185 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:09.849691 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:09.849693 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:09.849693 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:09.849700 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:09.851347 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:09.851965 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:09.851967 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:09.851967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:09.852041 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:09.852380 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:09.852382 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:09.852382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:09.852533 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:09.852597 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:09.852861 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 00:07:09.852971 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 00:07:09.853240 6950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:09.843279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45835 (local address 127.0.98.3:34633)
0620 00:07:09.843456 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:09.843501 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:09.843989 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:09.844431 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:09.844435 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:09.845090 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:09.845299 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:09.845304 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:09.846021 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:09.846022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:09.846856 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:09.846859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:09.847213 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:09.847213 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:09.847353 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:09.847988 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:09.847996 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:09.849580 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:09.851469 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:09.851477 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:09.851480 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:09.851869 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:09.852131 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:09.852132 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:09.852133 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:09.852279 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:09.852599 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:09.852603 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:09.852737 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 00:07:09.852826 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 00:07:09.852958 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":89,"threads_started":1}
I20260620 00:07:09.853765 6948 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45835
I20260620 00:07:09.853855 6948 heartbeater.cc:464] Registering TS with master...
I20260620 00:07:09.854000 6948 heartbeater.cc:511] Master 127.0.98.62:45835 requested a full tablet report, sending...
I20260620 00:07:09.854367 6489 ts_manager.cc:194] Registered new tserver with Master: 87af16a18180461ca7afc0bff09802ab (127.0.98.3:46179)
I20260620 00:07:09.854816 6489 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34633
I20260620 00:07:09.863570 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260620 00:07:09.897235 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33203
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45835,127.0.98.61:33203
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:07:09.981017 6953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:09.981256 6953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:09.981279 6953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:09.982563 6953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:07:09.982609 6953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:09.982625 6953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:09.982640 6953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:07:09.982651 6953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:07:09.984135 6953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45835,127.0.98.61:33203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
W20260620 00:07:09.984236 6953 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:45835,127.0.98.61:33203'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 00:07:09.984421 6953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:09.984656 6953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:09.990206 6958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:09.990217 6959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:09.990234 6961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:09.990443 6953 server_base.cc:1034] running on GCE node
I20260620 00:07:09.990698 6953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:09.991021 6953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:09.992225 6953 hybrid_clock.cc:648] HybridClock initialized: now 1781914029992208 us; error 48 us; skew 500 ppm
Jun 20 00:07:09 dist-test-slave-grqt krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914029, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:09.994968 6953 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260620 00:07:09.995534 6953 webserver.cc:460] Webserver started at http://127.0.98.61:45529/ using document root <none> and password file <none>
I20260620 00:07:09.995894 6953 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:09.995983 6953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:09.996169 6953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:09.997509 6953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-1/data/instance:
uuid: "58f4848318664a7586274a478661eed1"
format_stamp: "Formatted at 2026-06-20 00:07:09 on dist-test-slave-grqt"
server_key: "d9a8fb2c4a3a25bb6d0080f84078366a"
server_key_iv: "ed66c411fbe2f677d07db240b41d5657"
server_key_version: "encryptionkey@0"
I20260620 00:07:09.997911 6953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-1/wal/instance:
uuid: "58f4848318664a7586274a478661eed1"
format_stamp: "Formatted at 2026-06-20 00:07:09 on dist-test-slave-grqt"
server_key: "d9a8fb2c4a3a25bb6d0080f84078366a"
server_key_iv: "ed66c411fbe2f677d07db240b41d5657"
server_key_version: "encryptionkey@0"
I20260620 00:07:09.999343 6953 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:07:10.000351 6968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:10.000566 6953 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:10.000674 6953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-1/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-1/wal
uuid: "58f4848318664a7586274a478661eed1"
format_stamp: "Formatted at 2026-06-20 00:07:09 on dist-test-slave-grqt"
server_key: "d9a8fb2c4a3a25bb6d0080f84078366a"
server_key_iv: "ed66c411fbe2f677d07db240b41d5657"
server_key_version: "encryptionkey@0"
I20260620 00:07:10.000761 6953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:10.038434 6953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:10.040593 6953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:10.040772 6953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:10.044958 6953 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:33203
I20260620 00:07:10.044994 7020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:33203 every 8 connection(s)
I20260620 00:07:10.045446 6953 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-1/data/info.pb
I20260620 00:07:10.046538 7021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:10.048228 7021 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:07:10.052876 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 6953
Jun 20 00:07:10 dist-test-slave-grqt krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914029, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 00:07:10 dist-test-slave-grqt krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 00:07:10.059151 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.049036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39206 (local address 127.0.98.62:45835)
0620 00:07:10.049102 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:10.049104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:10.049352 (+ 248us) server_negotiation.cc:407] Connection header received
0620 00:07:10.049936 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:10.049937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:10.049949 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:10.050019 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:10.050716 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.051245 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.052339 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.052638 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.052742 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.055645 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:10.055647 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:10.055647 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:10.055655 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:10.057622 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.058034 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.058035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.058036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.058093 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.058380 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.058381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.058381 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.058539 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:10.058605 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:10.058994 (+ 389us) server_negotiation.cc:299] Negotiation successful
0620 00:07:10.059078 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 00:07:10.059377 7023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.049005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45835 (local address 127.0.0.1:39206)
0620 00:07:10.049208 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:10.049255 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:10.049821 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:10.050144 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:10.050148 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:10.050414 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:10.050615 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.050619 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.051338 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.051339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.052168 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.052170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.052768 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.052769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.052886 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.053699 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:10.053765 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:10.055498 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:10.057701 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.057707 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.057709 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.057956 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.058159 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.058160 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.058160 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.058265 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.058620 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:10.058626 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:10.058824 (+ 198us) client_negotiation.cc:769] Sending connection context
0620 00:07:10.058949 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 00:07:10.059124 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":119,"threads_started":1}
I20260620 00:07:10.060027 7021 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33203 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:07:10.062757 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.053419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34108 (local address 127.0.98.61:33203)
0620 00:07:10.053594 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:10.053599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:10.053696 (+ 97us) server_negotiation.cc:407] Connection header received
0620 00:07:10.053782 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:10.053785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:10.053810 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:10.053898 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:10.054588 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.055329 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.056290 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.056691 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.056789 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.059521 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:10.059526 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:10.059527 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:10.059533 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:10.061538 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.061965 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.061967 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.061968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.062020 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.062222 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.062223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.062223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.062347 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:10.062466 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:10.062564 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 00:07:10.062621 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":103,"threads_started":1}
Jun 20 00:07:10 dist-test-slave-grqt krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914029, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 00:07:10.068388 7023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.060322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33203 (local address 127.0.0.1:34122)
0620 00:07:10.060373 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:10.060384 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:10.060458 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:10.060640 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:10.060641 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:10.060806 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:10.060839 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.060840 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.061791 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.061793 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.062598 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.062600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.063096 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.063097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.063174 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.063587 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:10.063591 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:10.064788 (+ 1197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:10.067419 (+ 2631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.067420 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.067420 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.067594 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.067798 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.067799 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.067799 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.067851 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.068113 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:10.068113 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:10.068155 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 00:07:10.068232 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 00:07:10.068315 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 00:07:10.068569 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.060381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34122 (local address 127.0.98.61:33203)
0620 00:07:10.060518 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:10.060520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:10.060527 (+ 7us) server_negotiation.cc:407] Connection header received
0620 00:07:10.060546 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:10.060547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:10.060557 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:10.060626 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:10.060912 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.061716 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.062680 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.063018 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.063085 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.064895 (+ 1810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:10.064896 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:10.064896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:10.064903 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:10.067311 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.067674 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.067675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.067675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.067732 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.067893 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.067893 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.067893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.068000 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:10.068058 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:10.068287 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 00:07:10.068367 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
I20260620 00:07:10.070154 7021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1: Bootstrap starting.
I20260620 00:07:10.070828 7021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:10.071111 7021 log.cc:824] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1: Log is configured to *not* fsync() on all Append() calls
I20260620 00:07:10.071930 7021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1: No bootstrap required, opened a new log
I20260620 00:07:10.073467 7021 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33203 } }
I20260620 00:07:10.073619 7021 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:10.073665 7021 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58f4848318664a7586274a478661eed1, State: Initialized, Role: FOLLOWER
I20260620 00:07:10.073835 7021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33203 } }
I20260620 00:07:10.074182 7021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:07:10.074150 7028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33203 } } }
I20260620 00:07:10.074323 7028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 00:07:10.075829 7039 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:07:10.075888 7039 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:07:10.077600 6953 master_runner.cc:428] Detected that this master 58f4848318664a7586274a478661eed1 is joining an existing cluster
I20260620 00:07:10.077656 6953 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:33203
I20260620 00:07:10.085500 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.078668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45835 (local address 127.0.0.1:39214)
0620 00:07:10.078825 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:10.078836 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:10.078921 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:10.079095 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:10.079096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:10.079260 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:10.079337 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.079339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.080327 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.080328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.080903 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.080905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.081485 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.081487 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.081557 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.082060 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:10.082064 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:10.082538 (+ 474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:10.084286 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.084287 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.084287 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.084483 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.084707 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.084708 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.084708 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.084783 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.085090 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:10.085092 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:10.085152 (+ 60us) client_negotiation.cc:769] Sending connection context
0620 00:07:10.085229 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 00:07:10.085342 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":87,"threads_started":1}
I20260620 00:07:10.085855 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.078727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39214 (local address 127.0.98.62:45835)
0620 00:07:10.078770 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:10.078773 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:10.078892 (+ 119us) server_negotiation.cc:407] Connection header received
0620 00:07:10.079000 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:10.079001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:10.079013 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:10.079082 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:10.079423 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.080222 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.080990 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.081296 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.081424 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.082630 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:10.082632 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:10.082632 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:10.082639 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:10.084137 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.084577 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.084578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.084578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.084638 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.084850 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.084851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.084852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.085007 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:10.085080 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:10.085644 (+ 564us) server_negotiation.cc:299] Negotiation successful
0620 00:07:10.085743 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 00:07:10.088933 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.078936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34134 (local address 127.0.98.61:33203)
0620 00:07:10.079021 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:10.079022 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:10.081640 (+ 2618us) server_negotiation.cc:407] Connection header received
0620 00:07:10.082058 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:10.082059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:10.082070 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:10.082141 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:10.082533 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.083279 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.084291 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.084598 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.084638 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.085890 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:10.085891 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:10.085891 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:10.085897 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:10.087631 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.088064 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.088065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.088066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.088130 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.088373 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.088374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.088375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.088510 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:10.088580 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:10.088750 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 00:07:10.088833 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 00:07:10.089202 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.078894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33203 (local address 127.0.0.1:34134)
0620 00:07:10.081573 (+ 2679us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:10.081585 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:10.081664 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:10.082162 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:10.082163 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:10.082379 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:10.082443 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.082446 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.083380 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.083381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.084199 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.084202 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.084723 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.084723 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.084779 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.085245 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:10.085258 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:10.085811 (+ 553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:10.087742 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.087743 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.087744 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.087981 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.088211 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.088212 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.088212 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.088298 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.088602 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:10.088603 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:10.088637 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 00:07:10.089015 (+ 378us) client_negotiation.cc:240] Negotiation successful
0620 00:07:10.089107 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2608,"thread_start_us":140,"threads_started":1}
Jun 20 00:07:10 dist-test-slave-grqt krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914029, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 00:07:10.097128 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.088651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34144 (local address 127.0.98.61:33203)
0620 00:07:10.088694 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:10.088696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:10.088705 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:07:10.088818 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:10.088819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:10.088829 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:10.088874 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:10.089696 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.090268 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.091105 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.091346 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.091425 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.093738 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:10.093739 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:10.093740 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:10.093748 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:10.095596 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.096130 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.096131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.096131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.096197 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.096444 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.096445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.096445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.096615 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:10.096681 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:10.096897 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 00:07:10.097011 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 00:07:10.097144 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.088397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33203 (local address 127.0.0.1:34144)
0620 00:07:10.088577 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:10.088624 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:10.088693 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:10.088890 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:10.088894 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:10.089541 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:10.089612 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.089615 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.090361 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.090362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.090991 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.090994 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.091436 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.091437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.091545 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.092101 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:10.092106 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:10.093627 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:10.095706 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.095717 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.095720 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.096037 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.096283 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.096284 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.096284 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.096357 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.096686 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:10.096695 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:10.096759 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 00:07:10.096867 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 00:07:10.097014 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":127,"threads_started":1}
I20260620 00:07:10.097839 6489 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:33203: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "58f4848318664a7586274a478661eed1"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 33203
}
attrs {
promote: true
}
}
dest_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1"
cas_config_opid_index: -1
I20260620 00:07:10.098205 6489 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: true } }
I20260620 00:07:10.098775 7049 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 58f4848318664a7586274a478661eed1 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: true } } }
I20260620 00:07:10.099109 6995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:10.099974 7049 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:33203
I20260620 00:07:10.100121 6995 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [term 1 FOLLOWER]: Refusing update from remote peer 60dfbd8a9e3a4e4aaad18e7dac6d0ea1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 00:07:10.100255 6953 master.cc:537] Master@127.0.98.61:33203 shutting down...
I20260620 00:07:10.100287 7048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: true } } }
I20260620 00:07:10.100366 7048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [sys.catalog]: This master's current role is: LEADER
I20260620 00:07:10.100885 7048 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 00:07:10.101889 6473 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 00:07:10.102455 6473 consensus_peers.cc:487] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 -> Peer 58f4848318664a7586274a478661eed1 (127.0.98.61:33203): Couldn't send request to peer 58f4848318664a7586274a478661eed1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 00:07:10.103717 6953 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:07:10.103863 6953 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:07:10.103928 6953 tablet_replica.cc:330] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1: stopping tablet replica
I20260620 00:07:10.117198 6953 master.cc:552] Master@127.0.98.61:33203 shutdown complete.
I20260620 00:07:10.117272 6953 master_runner.cc:305] Clearing existing system tablet
I20260620 00:07:10.117445 6953 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:10.117487 6953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:10.118389 6953 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:07:10.119028 7053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:10.119343 6953 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:10.119411 6953 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:07:10.119438 6953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-1/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-1/wal
uuid: "58f4848318664a7586274a478661eed1"
format_stamp: "Formatted at 2026-06-20 00:07:09 on dist-test-slave-grqt"
server_key: "d9a8fb2c4a3a25bb6d0080f84078366a"
server_key_iv: "ed66c411fbe2f677d07db240b41d5657"
server_key_version: "encryptionkey@0"
I20260620 00:07:10.120366 6953 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:07:10.121985 6953 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 00:07:10.122071 6953 log.cc:1196] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 00:07:10.122277 6953 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1: Deleting consensus metadata
I20260620 00:07:10.122607 6953 master_runner.cc:315] Copying system tablet from 127.0.98.62:45835
I20260620 00:07:10.123293 6953 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:45835
I20260620 00:07:10.130271 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.123742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39216 (local address 127.0.98.62:45835)
0620 00:07:10.123805 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:10.123807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:10.123920 (+ 113us) server_negotiation.cc:407] Connection header received
0620 00:07:10.123994 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:10.123995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:10.124005 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:10.124055 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:10.124461 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.124989 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.126138 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.126343 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.126410 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.127552 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:10.127553 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:10.127553 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:10.127561 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:10.129131 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.129558 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.129559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.129560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.129613 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.129810 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.129810 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.129810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.129914 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:10.129960 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:10.130113 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 00:07:10.130173 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 00:07:10.130327 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.123648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45835 (local address 127.0.0.1:39216)
0620 00:07:10.123830 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:10.123843 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:10.123926 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:10.124072 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:10.124073 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:10.124274 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:10.124348 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.124350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.125080 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.125081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.126039 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.126042 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.126430 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.126431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.126501 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.127029 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:10.127034 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:10.127445 (+ 411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:10.129223 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.129225 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.129225 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.129481 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.129682 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.129682 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.129683 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.129749 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.129973 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:10.129974 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:10.130022 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 00:07:10.130098 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 00:07:10.130207 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":109,"threads_started":1}
I20260620 00:07:10.130677 6509 tablet_copy_service.cc:140] P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 58f4848318664a7586274a478661eed1 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:39216)
I20260620 00:07:10.130791 6509 tablet_copy_service.cc:161] P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 58f4848318664a7586274a478661eed1 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:39216: session id = 58f4848318664a7586274a478661eed1-00000000000000000000000000000000
I20260620 00:07:10.131659 6509 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 00:07:10.132310 6953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 00:07:10.133327 6953 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1: tablet copy: Starting download of 0 data blocks...
I20260620 00:07:10.133528 6953 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1: tablet copy: Starting download of 1 WAL segments...
I20260620 00:07:10.134474 6953 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 00:07:10.135340 6509 tablet_copy_service.cc:342] P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1: Request end of tablet copy session 58f4848318664a7586274a478661eed1-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:39216
I20260620 00:07:10.135406 6509 tablet_copy_service.cc:433] P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1: ending tablet copy session 58f4848318664a7586274a478661eed1-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 58f4848318664a7586274a478661eed1
I20260620 00:07:10.137733 6953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:10.139009 7061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:10.139058 7063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:10.139155 7060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:10.139369 6953 server_base.cc:1034] running on GCE node
I20260620 00:07:10.139461 6953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:10.139593 6953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:10.140723 6953 hybrid_clock.cc:648] HybridClock initialized: now 1781914030140715 us; error 35 us; skew 500 ppm
Jun 20 00:07:10 dist-test-slave-grqt krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914030, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:10.142082 6953 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260620 00:07:10.142407 6953 webserver.cc:460] Webserver started at http://127.0.98.61:37541/ using document root <none> and password file <none>
I20260620 00:07:10.142529 6953 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:10.142594 6953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:10.143227 6953 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260620 00:07:10.143796 7070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:10.143919 6953 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 00:07:10.143967 6953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-1/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-1/wal
uuid: "58f4848318664a7586274a478661eed1"
format_stamp: "Formatted at 2026-06-20 00:07:09 on dist-test-slave-grqt"
server_key: "d9a8fb2c4a3a25bb6d0080f84078366a"
server_key_iv: "ed66c411fbe2f677d07db240b41d5657"
server_key_version: "encryptionkey@0"
I20260620 00:07:10.144030 6953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:10.152732 6953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:10.154501 6953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:10.157709 6953 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:33203
I20260620 00:07:10.158037 7122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:33203 every 8 connection(s)
I20260620 00:07:10.158124 6953 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-1/data/info.pb
I20260620 00:07:10.158885 7123 sys_catalog.cc:263] Verifying existing consensus state
I20260620 00:07:10.159309 7123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1: Bootstrap starting.
I20260620 00:07:10.161690 7123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 00:07:10.161942 7123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1: Bootstrap complete.
I20260620 00:07:10.162145 7123 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: true } }
I20260620 00:07:10.162215 7123 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 58f4848318664a7586274a478661eed1, State: Initialized, Role: LEARNER
I20260620 00:07:10.162241 7123 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: true } }
I20260620 00:07:10.162444 7123 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:07:10.162462 7126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: true } } }
I20260620 00:07:10.162572 7126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [sys.catalog]: This master's current role is: LEARNER
I20260620 00:07:10.163976 7137 catalog_manager.cc:1259] Loaded cluster ID: 7bcc619fd37045da9bc9c98085719301
I20260620 00:07:10.164054 7137 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1: loading cluster ID for follower catalog manager: success
I20260620 00:07:10.169090 7137 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1: acquiring CA information for follower catalog manager: success
I20260620 00:07:10.169502 7137 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 00:07:10.176209 7138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.164967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34164 (local address 127.0.98.61:33203)
0620 00:07:10.165087 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:10.165089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:10.167914 (+ 2825us) server_negotiation.cc:407] Connection header received
0620 00:07:10.167995 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:10.167996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:10.168010 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:10.168890 (+ 880us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:10.169145 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.170200 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.171253 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.171544 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.172123 (+ 579us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.173209 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:10.173210 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:10.173210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:10.173217 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:10.175004 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.175416 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.175417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.175417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.175472 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.175679 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.175679 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.175680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.175785 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:10.175888 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:10.176019 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 00:07:10.176098 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":53,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260620 00:07:10.205087 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35537
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45835,127.0.98.61:33203,127.0.98.60:35537
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:07:10.292433 7141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:10.292716 7141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:10.292757 7141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:10.294088 7141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:07:10.294167 7141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:10.294195 7141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:10.294229 7141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:07:10.294250 7141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:07:10.295818 7141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45835,127.0.98.61:33203,127.0.98.60:35537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:10.296160 7141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:10.296418 7141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:10.301926 7149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:10.301940 7146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:10.301930 7147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:10.302040 7141 server_base.cc:1034] running on GCE node
I20260620 00:07:10.302430 7141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:10.302747 7141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:10.303917 7141 hybrid_clock.cc:648] HybridClock initialized: now 1781914030303913 us; error 40 us; skew 500 ppm
Jun 20 00:07:10 dist-test-slave-grqt krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914030, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:10.306951 7141 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260620 00:07:10.307488 7141 webserver.cc:460] Webserver started at http://127.0.98.60:39233/ using document root <none> and password file <none>
I20260620 00:07:10.307704 7141 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:10.307747 7141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:10.307843 7141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:10.308902 7141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-2/data/instance:
uuid: "b410653b3fa54e709136276f3e32a9bb"
format_stamp: "Formatted at 2026-06-20 00:07:10 on dist-test-slave-grqt"
server_key: "71431766122e32b46fc1052cd287f820"
server_key_iv: "b1186c59fc109e30d63e003cf23f1bc5"
server_key_version: "encryptionkey@0"
I20260620 00:07:10.309221 7141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-2/wal/instance:
uuid: "b410653b3fa54e709136276f3e32a9bb"
format_stamp: "Formatted at 2026-06-20 00:07:10 on dist-test-slave-grqt"
server_key: "71431766122e32b46fc1052cd287f820"
server_key_iv: "b1186c59fc109e30d63e003cf23f1bc5"
server_key_version: "encryptionkey@0"
I20260620 00:07:10.310803 7141 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:07:10.311510 7156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:10.311694 7141 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 00:07:10.311769 7141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-2/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-2/wal
uuid: "b410653b3fa54e709136276f3e32a9bb"
format_stamp: "Formatted at 2026-06-20 00:07:10 on dist-test-slave-grqt"
server_key: "71431766122e32b46fc1052cd287f820"
server_key_iv: "b1186c59fc109e30d63e003cf23f1bc5"
server_key_version: "encryptionkey@0"
I20260620 00:07:10.311902 7141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:10.325899 7141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:10.328445 7141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:10.328601 7141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:10.333070 7141 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35537
I20260620 00:07:10.333143 7208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35537 every 8 connection(s)
I20260620 00:07:10.333572 7141 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-2/data/info.pb
I20260620 00:07:10.334597 7209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:10.336272 7209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:07:10.340776 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 7141
Jun 20 00:07:10 dist-test-slave-grqt krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914030, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 00:07:10.346447 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.337187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39218 (local address 127.0.98.62:45835)
0620 00:07:10.337263 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:10.337265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:10.337465 (+ 200us) server_negotiation.cc:407] Connection header received
0620 00:07:10.337967 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:10.337968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:10.337978 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:10.338042 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:10.338737 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.339238 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.340252 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.340440 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.340496 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.343088 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:10.343090 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:10.343090 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:10.343098 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:10.344635 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.345164 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.345165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.345166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.345238 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.345713 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.345714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.345715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.345879 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:10.345959 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:10.346230 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 00:07:10.346324 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 00:07:10.346561 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.337062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45835 (local address 127.0.0.1:39218)
0620 00:07:10.337251 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:10.337328 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:10.337880 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:10.338137 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:10.338141 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:10.338434 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:10.338633 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.338637 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.339340 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.339340 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.340170 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.340172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.340518 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.340519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.340624 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.341256 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:10.341265 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:10.342986 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:10.344721 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.344726 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.344728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.345053 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.345361 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.345361 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.345362 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.345630 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.345962 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:10.345967 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:10.346089 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 00:07:10.346165 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 00:07:10.346319 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":107,"threads_started":1}
Jun 20 00:07:10 dist-test-slave-grqt krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 00:07:10.347677 7209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33203 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:07:10.352026 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.341344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45758 (local address 127.0.98.60:35537)
0620 00:07:10.341576 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:10.341581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:10.341604 (+ 23us) server_negotiation.cc:407] Connection header received
0620 00:07:10.341623 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:10.341624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:10.341640 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:10.341725 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:10.342443 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.343210 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.344271 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.344574 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.344633 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.348383 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:10.348394 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:10.348395 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:10.348402 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:10.350413 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.350833 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.350836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.350837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.350890 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.351410 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.351411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.351411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.351550 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:10.351682 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:10.351825 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 00:07:10.351910 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":151,"threads_started":1}
Jun 20 00:07:10 dist-test-slave-grqt krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914030, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 00:07:10.355353 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.348053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33203 (local address 127.0.0.1:34174)
0620 00:07:10.348104 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:10.348116 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:10.348229 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:10.348376 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:10.348376 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:10.348534 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:10.348581 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.348583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.349671 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.349672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.350167 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.350169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.350691 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.350692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.350802 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.351215 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:10.351220 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:10.352787 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:10.354502 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.354503 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.354503 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.354659 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.354886 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.354886 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.354887 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.354936 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.355119 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:10.355120 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:10.355154 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 00:07:10.355199 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 00:07:10.355289 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 00:07:10.355358 7138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.348129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34174 (local address 127.0.98.61:33203)
0620 00:07:10.348194 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:10.348196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:10.348206 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:07:10.348307 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:10.348309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:10.348322 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:10.348373 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:10.348698 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.349515 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.350271 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.350601 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.350656 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.352872 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:10.352874 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:10.352874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:10.352882 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:10.354420 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.354751 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.354752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.354753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.354809 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.354976 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.354976 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.354976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.355071 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:10.355118 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:10.355227 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 00:07:10.355297 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 00:07:10.355840 7209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35537 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 00:07:10 dist-test-slave-grqt krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914030, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 00:07:10.362231 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.356124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35537 (local address 127.0.0.1:45766)
0620 00:07:10.356176 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:10.356188 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:10.356243 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:10.356347 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:10.356348 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:10.356484 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:10.356519 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.356520 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.357125 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.357126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.357702 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.357704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.358075 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.358076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.358116 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.358570 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:10.358575 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:10.359798 (+ 1223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:10.361221 (+ 1423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.361222 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.361222 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.361451 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.361653 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.361654 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.361654 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.361704 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.361918 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:10.361919 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:10.361957 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 00:07:10.362025 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 00:07:10.362107 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 00:07:10.362231 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.356158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45766 (local address 127.0.98.60:35537)
0620 00:07:10.356190 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:10.356192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:10.356228 (+ 36us) server_negotiation.cc:407] Connection header received
0620 00:07:10.356298 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:10.356298 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:10.356307 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:10.356343 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:10.356576 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.357037 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.357804 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.357988 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.358063 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.359880 (+ 1817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:10.359881 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:10.359882 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:10.359887 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:10.361141 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.361522 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.361523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.361524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.361577 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.361749 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.361750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.361750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.361842 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:10.361889 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:10.362042 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 00:07:10.362153 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":15}
I20260620 00:07:10.364073 7209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb: Bootstrap starting.
I20260620 00:07:10.364982 7209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:10.365286 7209 log.cc:824] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb: Log is configured to *not* fsync() on all Append() calls
I20260620 00:07:10.366139 7209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb: No bootstrap required, opened a new log
I20260620 00:07:10.368022 7209 raft_consensus.cc:348] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33203 } } peers { permanent_uuid: "b410653b3fa54e709136276f3e32a9bb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35537 } }
I20260620 00:07:10.368160 7209 raft_consensus.cc:374] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:10.368183 7209 raft_consensus.cc:729] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b410653b3fa54e709136276f3e32a9bb, State: Initialized, Role: FOLLOWER
I20260620 00:07:10.368281 7209 consensus_queue.cc:260] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33203 } } peers { permanent_uuid: "b410653b3fa54e709136276f3e32a9bb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35537 } }
I20260620 00:07:10.368568 7214 sys_catalog.cc:455] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33203 } } peers { permanent_uuid: "b410653b3fa54e709136276f3e32a9bb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35537 } } }
I20260620 00:07:10.368608 7209 sys_catalog.cc:564] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:07:10.368682 7214 sys_catalog.cc:458] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [sys.catalog]: This master's current role is: FOLLOWER
W20260620 00:07:10.370565 7225 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:07:10.370632 7225 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:07:10.373126 7141 master_runner.cc:428] Detected that this master b410653b3fa54e709136276f3e32a9bb is joining an existing cluster
I20260620 00:07:10.373172 7141 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:35537
I20260620 00:07:10.382385 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.374636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39222 (local address 127.0.98.62:45835)
0620 00:07:10.374696 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:10.374699 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:10.374796 (+ 97us) server_negotiation.cc:407] Connection header received
0620 00:07:10.375023 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:10.375024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:10.375038 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:10.375071 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:10.375694 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.376475 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.377797 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.378131 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.378181 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.379517 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:10.379518 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:10.379519 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:10.379526 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:10.381117 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.381653 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.381654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.381654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.381714 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.381926 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.381926 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.381926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.382029 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:10.382079 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:10.382253 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 00:07:10.382317 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 00:07:10.382804 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.374534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45835 (local address 127.0.0.1:39222)
0620 00:07:10.374715 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:10.374733 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:10.374834 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:10.375174 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:10.375175 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:10.375467 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:10.375588 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.375591 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.376582 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.376582 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.377686 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.377689 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.378308 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.378308 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.378369 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.378872 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:10.378878 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:10.379397 (+ 519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:10.381243 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.381245 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.381245 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.381561 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.381788 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.381789 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.381790 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.381861 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.382092 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:10.382093 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:10.382155 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 00:07:10.382236 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 00:07:10.382355 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":109,"threads_started":1}
Jun 20 00:07:10 dist-test-slave-grqt krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914029, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 00:07:10.401794 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.385220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35537 (local address 127.0.0.1:45776)
0620 00:07:10.385273 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:10.385282 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:10.385322 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:10.385620 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:10.385621 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:10.385785 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:10.385827 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.385830 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.386603 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.386604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.387127 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.387129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.387539 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.387540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.387605 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.388061 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:10.388066 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:10.389559 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:10.400166 (+ 10607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.400168 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.400169 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.400413 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.400680 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.400681 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.400681 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.400744 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.401428 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:10.401429 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:10.401482 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 00:07:10.401533 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 00:07:10.401664 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 00:07:10.401969 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.375004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45772 (local address 127.0.98.60:35537)
0620 00:07:10.375124 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:10.375125 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:10.375196 (+ 71us) server_negotiation.cc:407] Connection header received
0620 00:07:10.375293 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:10.375293 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:10.375303 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:10.375336 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:10.375862 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.376683 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.377964 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.378269 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.378312 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.398853 (+ 20541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:10.398855 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:10.398855 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:10.398863 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:10.400491 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.400858 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.400859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.400860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.401236 (+ 376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.401449 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.401450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.401450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.401565 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:10.401604 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:10.401797 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 00:07:10.401875 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260620 00:07:10.402268 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.374977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35537 (local address 127.0.0.1:45772)
0620 00:07:10.375135 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:10.375144 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:10.375220 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:10.375401 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:10.375401 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:10.375693 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:10.375767 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.375769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.376775 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.376775 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.377853 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.377857 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.378392 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.378393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.378461 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.398088 (+ 19627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:10.398095 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:10.398688 (+ 593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:10.400588 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.400589 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.400589 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.400787 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.401311 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.401311 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.401312 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.401363 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.401642 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:10.401642 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:10.401703 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 00:07:10.402085 (+ 382us) client_negotiation.cc:240] Negotiation successful
0620 00:07:10.402172 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":93,"thread_start_us":108,"threads_started":1}
I20260620 00:07:10.402961 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.374781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33203 (local address 127.0.0.1:34184)
0620 00:07:10.375092 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:10.375103 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:10.375274 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:10.375477 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:10.375478 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:10.375707 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:10.375809 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.375811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.376879 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.376880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.377881 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.377883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.378476 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.378476 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.378556 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.398468 (+ 19912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:10.398473 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:10.399496 (+ 1023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:10.401883 (+ 2387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.401885 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.401885 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.402087 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.402313 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.402314 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.402314 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.402379 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.402633 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:10.402633 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:10.402682 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 00:07:10.402757 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 00:07:10.402854 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":75,"threads_started":1}
I20260620 00:07:10.403244 7138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.374841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34184 (local address 127.0.98.61:33203)
0620 00:07:10.374890 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:10.374892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:10.375246 (+ 354us) server_negotiation.cc:407] Connection header received
0620 00:07:10.375358 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:10.375360 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:10.375372 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:10.375407 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:10.375904 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.376763 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.377978 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.378334 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.378379 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.399621 (+ 21242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:10.399623 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:10.399623 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:10.399633 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:10.401792 (+ 2159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.402175 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.402176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.402177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.402235 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.402437 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.402438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.402438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.402555 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:10.402628 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:10.403069 (+ 441us) server_negotiation.cc:299] Negotiation successful
0620 00:07:10.403152 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 00:07:10.403661 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.385289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45776 (local address 127.0.98.60:35537)
0620 00:07:10.385491 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:10.385492 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:10.385498 (+ 6us) server_negotiation.cc:407] Connection header received
0620 00:07:10.385529 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:10.385529 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:10.385538 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:10.385616 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:10.385923 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.386504 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.387225 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.387462 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.387543 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.389654 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:10.389655 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:10.389656 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:10.389670 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:10.399771 (+ 10101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.400518 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.400519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.400520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.400580 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.400823 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.400824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.400824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.401301 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:10.402861 (+ 1560us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:10.402897 (+ 36us) server_negotiation.cc:299] Negotiation successful
0620 00:07:10.403551 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":135,"threads_started":1}
I20260620 00:07:10.403728 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 6551
I20260620 00:07:10.409675 6489 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:35537: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b410653b3fa54e709136276f3e32a9bb"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 35537
}
attrs {
promote: true
}
}
dest_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1"
cas_config_opid_index: 5
I20260620 00:07:10.409747 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44335
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45835
--webserver_port=40623
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:45835,127.0.98.61:33203
--tserver_master_addrs=127.0.98.62:45835,127.0.98.61:33203,127.0.98.60:35537 with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:07:10.409919 6489 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: true } } peers { permanent_uuid: "b410653b3fa54e709136276f3e32a9bb" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35537 } attrs { promote: true } }
I20260620 00:07:10.410281 7048 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b410653b3fa54e709136276f3e32a9bb (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: true } } peers { permanent_uuid: "b410653b3fa54e709136276f3e32a9bb" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35537 } attrs { promote: true } } }
I20260620 00:07:10.411223 7048 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:35537
I20260620 00:07:10.411314 7048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: true } } peers { permanent_uuid: "b410653b3fa54e709136276f3e32a9bb" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35537 } attrs { promote: true } } }
I20260620 00:07:10.411384 7048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [sys.catalog]: This master's current role is: LEADER
I20260620 00:07:10.411831 7141 master.cc:537] Master@127.0.98.60:35537 shutting down...
W20260620 00:07:10.413703 6473 consensus_peers.cc:487] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 -> Peer b410653b3fa54e709136276f3e32a9bb (127.0.98.60:35537): Couldn't send request to peer b410653b3fa54e709136276f3e32a9bb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 00:07:10.415812 7141 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [term 0 FOLLOWER]: Raft consensus shutting down.
I20260620 00:07:10.415920 7141 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [term 0 FOLLOWER]: Raft consensus is shut down!
I20260620 00:07:10.415980 7141 tablet_replica.cc:330] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb: stopping tablet replica
I20260620 00:07:10.416528 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.411634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33203 (local address 127.0.0.1:34186)
0620 00:07:10.411818 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:10.411831 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:10.411888 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:10.412125 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:10.412126 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:10.412128 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:07:10.412175 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.412177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.413127 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.413128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.415053 (+ 1925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.415058 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.416196 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.416198 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.416280 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.416311 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 00:07:10.416370 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 00:07:10.416430 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168}
I20260620 00:07:10.416841 7138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.411745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34186 (local address 127.0.98.61:33203)
0620 00:07:10.411787 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:10.411790 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:10.411940 (+ 150us) server_negotiation.cc:407] Connection header received
0620 00:07:10.411979 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:10.411980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:10.412003 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:10.412057 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:07:10.412245 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.413024 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.415174 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.415986 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.416065 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.416154 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:10.416700 (+ 546us) server_negotiation.cc:299] Negotiation successful
0620 00:07:10.416751 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 00:07:10.417076 7097 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [term 1 LEARNER]: Refusing update from remote peer 60dfbd8a9e3a4e4aaad18e7dac6d0ea1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 00:07:10.417302 7049 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 00:07:10.417989 7097 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b410653b3fa54e709136276f3e32a9bb (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: true } } peers { permanent_uuid: "b410653b3fa54e709136276f3e32a9bb" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35537 } attrs { promote: true } } }
I20260620 00:07:10.419270 7126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60dfbd8a9e3a4e4aaad18e7dac6d0ea1. Latest consensus state: current_term: 1 leader_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: true } } peers { permanent_uuid: "b410653b3fa54e709136276f3e32a9bb" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35537 } attrs { promote: true } } }
I20260620 00:07:10.419364 7126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [sys.catalog]: This master's current role is: LEARNER
I20260620 00:07:10.419317 7237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: true } } peers { permanent_uuid: "b410653b3fa54e709136276f3e32a9bb" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35537 } attrs { promote: true } } }
I20260620 00:07:10.419423 7237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [sys.catalog]: This master's current role is: LEARNER
I20260620 00:07:10.420099 7049 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1: attempting to promote NON_VOTER 58f4848318664a7586274a478661eed1 to VOTER
I20260620 00:07:10.420356 7049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "b410653b3fa54e709136276f3e32a9bb" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35537 } attrs { promote: true } }
I20260620 00:07:10.420969 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.420840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35537 (local address 127.0.0.1:45782)
0620 00:07:10.420881 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:10.420931 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260620 00:07:10.421016 7097 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [term 1 LEARNER]: Refusing update from remote peer 60dfbd8a9e3a4e4aaad18e7dac6d0ea1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 00:07:10.421344 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.421212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35537 (local address 127.0.0.1:45786)
0620 00:07:10.421285 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:10.421311 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 00:07:10.421451 6473 consensus_peers.cc:487] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 -> Peer b410653b3fa54e709136276f3e32a9bb (127.0.98.60:35537): Couldn't send request to peer b410653b3fa54e709136276f3e32a9bb. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 00:07:10.421622 7238 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 00:07:10.422266 7048 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 58f4848318664a7586274a478661eed1 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "b410653b3fa54e709136276f3e32a9bb" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35537 } attrs { promote: true } } }
I20260620 00:07:10.423164 7048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "b410653b3fa54e709136276f3e32a9bb" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35537 } attrs { promote: true } } }
I20260620 00:07:10.423233 7048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [sys.catalog]: This master's current role is: LEADER
I20260620 00:07:10.423277 7048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "b410653b3fa54e709136276f3e32a9bb" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35537 } attrs { promote: true } } }
I20260620 00:07:10.423313 7048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [sys.catalog]: This master's current role is: LEADER
I20260620 00:07:10.423597 7097 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 58f4848318664a7586274a478661eed1 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "b410653b3fa54e709136276f3e32a9bb" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35537 } attrs { promote: true } } }
I20260620 00:07:10.424599 7237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "b410653b3fa54e709136276f3e32a9bb" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35537 } attrs { promote: true } } }
I20260620 00:07:10.424695 7237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:07:10.430307 7141 master.cc:552] Master@127.0.98.60:35537 shutdown complete.
I20260620 00:07:10.430364 7141 master_runner.cc:305] Clearing existing system tablet
I20260620 00:07:10.430500 7141 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:10.430545 7141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:10.431563 7141 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 00:07:10.432160 7240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:10.432279 7141 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:10.432328 7141 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 00:07:10.432350 7141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-2/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-2/wal
uuid: "b410653b3fa54e709136276f3e32a9bb"
format_stamp: "Formatted at 2026-06-20 00:07:10 on dist-test-slave-grqt"
server_key: "71431766122e32b46fc1052cd287f820"
server_key_iv: "b1186c59fc109e30d63e003cf23f1bc5"
server_key_version: "encryptionkey@0"
I20260620 00:07:10.433197 7141 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:07:10.434964 7141 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 00:07:10.435042 7141 log.cc:1196] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 00:07:10.435242 7141 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb: Deleting consensus metadata
I20260620 00:07:10.435348 7141 master_runner.cc:315] Copying system tablet from 127.0.98.62:45835
I20260620 00:07:10.435951 7141 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:45835
I20260620 00:07:10.443434 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.436346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39236 (local address 127.0.98.62:45835)
0620 00:07:10.436393 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:10.436395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:10.436452 (+ 57us) server_negotiation.cc:407] Connection header received
0620 00:07:10.436525 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:10.436526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:10.436537 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:10.436614 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:10.437008 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.437800 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.438887 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.439182 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.439255 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.440490 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:10.440492 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:10.440492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:10.440500 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:10.442205 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.442616 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.442618 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.442618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.442687 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.442890 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.442891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.442891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.443028 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:10.443098 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:10.443258 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 00:07:10.443341 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 00:07:10.443444 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.436275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45835 (local address 127.0.0.1:39236)
0620 00:07:10.436394 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:10.436406 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:10.436468 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:10.436635 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:10.436635 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:10.436837 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:10.436917 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.436920 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.437897 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.437898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.438782 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.438784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.439269 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.439270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.439339 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.439893 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:10.439898 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:10.440384 (+ 486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:10.442302 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.442303 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.442304 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.442534 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.442759 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.442760 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.442761 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.442824 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.443096 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:10.443097 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:10.443145 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 00:07:10.443222 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 00:07:10.443333 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":65,"threads_started":1}
I20260620 00:07:10.444166 6509 tablet_copy_service.cc:140] P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b410653b3fa54e709136276f3e32a9bb ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:39236)
I20260620 00:07:10.444248 6509 tablet_copy_service.cc:161] P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b410653b3fa54e709136276f3e32a9bb at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:39236: session id = b410653b3fa54e709136276f3e32a9bb-00000000000000000000000000000000
I20260620 00:07:10.444833 6509 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 00:07:10.445307 7141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 00:07:10.446815 7141 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb: tablet copy: Starting download of 0 data blocks...
I20260620 00:07:10.447055 7141 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb: tablet copy: Starting download of 1 WAL segments...
I20260620 00:07:10.447976 7141 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 00:07:10.449018 6509 tablet_copy_service.cc:342] P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1: Request end of tablet copy session b410653b3fa54e709136276f3e32a9bb-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:39236
I20260620 00:07:10.449074 6509 tablet_copy_service.cc:433] P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1: ending tablet copy session b410653b3fa54e709136276f3e32a9bb-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b410653b3fa54e709136276f3e32a9bb
I20260620 00:07:10.451640 7141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:10.453248 7247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:10.454531 7250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:10.454603 7141 server_base.cc:1034] running on GCE node
W20260620 00:07:10.458357 7248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:10.458606 7141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:10.458778 7141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:10.459928 7141 hybrid_clock.cc:648] HybridClock initialized: now 1781914030459900 us; error 47 us; skew 500 ppm
Jun 20 00:07:10 dist-test-slave-grqt krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914030, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:10.461511 7141 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260620 00:07:10.461820 7141 webserver.cc:460] Webserver started at http://127.0.98.60:37709/ using document root <none> and password file <none>
I20260620 00:07:10.461942 7141 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:10.461977 7141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:10.462728 7141 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:07:10.463353 7257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:10.463505 7141 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 00:07:10.463554 7141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-2/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-2/wal
uuid: "b410653b3fa54e709136276f3e32a9bb"
format_stamp: "Formatted at 2026-06-20 00:07:10 on dist-test-slave-grqt"
server_key: "71431766122e32b46fc1052cd287f820"
server_key_iv: "b1186c59fc109e30d63e003cf23f1bc5"
server_key_version: "encryptionkey@0"
I20260620 00:07:10.463609 7141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:10.487435 7141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:10.489861 7141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:10.493160 7141 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35537
I20260620 00:07:10.493400 7309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35537 every 8 connection(s)
I20260620 00:07:10.493564 7141 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/master-2/data/info.pb
I20260620 00:07:10.494468 7310 sys_catalog.cc:263] Verifying existing consensus state
I20260620 00:07:10.494984 7310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb: Bootstrap starting.
I20260620 00:07:10.498514 7310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 00:07:10.498769 7310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb: Bootstrap complete.
I20260620 00:07:10.498980 7310 raft_consensus.cc:348] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "b410653b3fa54e709136276f3e32a9bb" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35537 } attrs { promote: true } }
I20260620 00:07:10.499075 7310 raft_consensus.cc:729] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b410653b3fa54e709136276f3e32a9bb, State: Initialized, Role: LEARNER
I20260620 00:07:10.499111 7310 consensus_queue.cc:260] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "b410653b3fa54e709136276f3e32a9bb" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35537 } attrs { promote: true } }
I20260620 00:07:10.499302 7310 sys_catalog.cc:564] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:07:10.499317 7313 sys_catalog.cc:455] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "b410653b3fa54e709136276f3e32a9bb" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35537 } attrs { promote: true } } }
I20260620 00:07:10.499531 7313 sys_catalog.cc:458] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [sys.catalog]: This master's current role is: LEARNER
I20260620 00:07:10.501569 7324 catalog_manager.cc:1259] Loaded cluster ID: 7bcc619fd37045da9bc9c98085719301
I20260620 00:07:10.501682 7324 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb: loading cluster ID for follower catalog manager: success
I20260620 00:07:10.502808 7324 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb: acquiring CA information for follower catalog manager: success
I20260620 00:07:10.503209 7324 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 00:07:10.517854 7235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:10.518110 7235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:10.518150 7235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:10.519347 7235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:10.519421 7235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:10.519470 7235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:07:10.521149 7235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44335
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40623
--tserver_master_addrs=127.0.98.62:45835,127.0.98.61:33203,127.0.98.60:35537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:10.521485 7235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:10.521747 7235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:10.527835 7330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:10.527933 7235 server_base.cc:1034] running on GCE node
W20260620 00:07:10.528030 7332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:10.527868 7329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:10.528296 7235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:10.528633 7235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:10.529803 7235 hybrid_clock.cc:648] HybridClock initialized: now 1781914030529778 us; error 42 us; skew 500 ppm
Jun 20 00:07:10 dist-test-slave-grqt krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914030, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:10.532487 7235 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 00:07:10.532976 7235 webserver.cc:460] Webserver started at http://127.0.98.1:40623/ using document root <none> and password file <none>
I20260620 00:07:10.533164 7235 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:10.533216 7235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:10.534690 7235 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:10.535281 7339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:10.535490 7235 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:10.535594 7235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-0/wal
uuid: "ee53defd13044c57bbb138930c23dbf6"
format_stamp: "Formatted at 2026-06-20 00:07:09 on dist-test-slave-grqt"
server_key: "f6cc9eb0d7ad8123230204a56051cadb"
server_key_iv: "cd46cb4ad8bec409f36adf33a20e9c63"
server_key_version: "encryptionkey@0"
I20260620 00:07:10.535977 7235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:10.553124 7235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:10.555915 7235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:10.556111 7235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:10.556375 7235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:07:10.556707 7235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:07:10.556764 7235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:10.556811 7235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:07:10.556838 7235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:10.563428 7235 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44335
I20260620 00:07:10.563580 7452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44335 every 8 connection(s)
I20260620 00:07:10.563906 7235 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:07:10.570911 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 7235
I20260620 00:07:10.571074 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 6685
Jun 20 00:07:10 dist-test-slave-grqt krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914030, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 20 00:07:10 dist-test-slave-grqt krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914030, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 20 00:07:10 dist-test-slave-grqt krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914030, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 00:07:10.577239 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.565651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53245 (local address 127.0.98.60:35537)
0620 00:07:10.565769 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:10.565771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:10.565778 (+ 7us) server_negotiation.cc:407] Connection header received
0620 00:07:10.566323 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:10.566324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:10.566335 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:10.566403 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:10.567076 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.568054 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.568871 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.569091 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.569175 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.573139 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:10.573141 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:10.573141 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:10.573149 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:10.575327 (+ 2178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.575963 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.575965 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.575965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.576039 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.576401 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.576402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.576402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.576571 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:10.576637 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:10.577014 (+ 377us) server_negotiation.cc:299] Negotiation successful
0620 00:07:10.577130 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":64,"threads_started":1}
I20260620 00:07:10.577527 7459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.565595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35537 (local address 127.0.98.1:53245)
0620 00:07:10.565703 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:10.565710 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:10.565972 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:10.566415 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:10.566415 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:10.566775 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:10.566971 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.566977 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.568132 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.568133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.568768 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.568770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.569184 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.569185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.569320 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.570014 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:10.570022 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:10.573031 (+ 3009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:10.575468 (+ 2437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.575479 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.575481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.575890 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.576141 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.576141 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.576142 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.576337 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.576645 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:10.576653 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:10.576857 (+ 204us) client_negotiation.cc:769] Sending connection context
0620 00:07:10.576977 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 00:07:10.577181 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":23,"thread_start_us":63,"threads_started":1}
I20260620 00:07:10.578001 7454 heartbeater.cc:347] Connected to a master server at 127.0.98.60:35537
I20260620 00:07:10.578104 7454 heartbeater.cc:464] Registering TS with master...
I20260620 00:07:10.578305 7454 heartbeater.cc:511] Master 127.0.98.60:35537 requested a full tablet report, sending...
I20260620 00:07:10.578627 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34161
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45835
--webserver_port=42847
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:45835,127.0.98.61:33203
--tserver_master_addrs=127.0.98.62:45835,127.0.98.61:33203,127.0.98.60:35537 with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:07:10.578926 7274 ts_manager.cc:194] Registered new tserver with Master: ee53defd13044c57bbb138930c23dbf6 (127.0.98.1:44335)
I20260620 00:07:10.579520 7458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.565430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33203 (local address 127.0.98.1:41191)
0620 00:07:10.565554 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:10.565565 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:10.565973 (+ 408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:10.566305 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:10.566306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:10.566740 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:10.566971 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.566977 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.567904 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.567905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.568775 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.568777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.569294 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.569295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.569385 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.570086 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:10.570091 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:10.572484 (+ 2393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:10.578214 (+ 5730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.578214 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.578215 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.578576 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.578810 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.578811 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.578812 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.578868 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.579164 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:10.579165 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:10.579210 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 00:07:10.579280 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 00:07:10.579396 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":18,"thread_start_us":63,"threads_started":1}
I20260620 00:07:10.579907 7138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.565479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41191 (local address 127.0.98.61:33203)
0620 00:07:10.565524 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:10.565526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:10.565620 (+ 94us) server_negotiation.cc:407] Connection header received
0620 00:07:10.566204 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:10.566205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:10.566215 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:10.566267 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:10.567076 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.567823 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.568874 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.569177 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.569246 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.572622 (+ 3376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:10.572623 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:10.572624 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:10.572633 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:10.574927 (+ 2294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.578658 (+ 3731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.578660 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.578660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.578729 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.578925 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.578925 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.578925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.579075 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:10.579138 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:10.579721 (+ 583us) server_negotiation.cc:299] Negotiation successful
0620 00:07:10.579820 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 00:07:10.580150 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.565135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35577 (local address 127.0.98.62:45835)
0620 00:07:10.565185 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:10.565188 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:10.565394 (+ 206us) server_negotiation.cc:407] Connection header received
0620 00:07:10.566063 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:10.566064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:10.566077 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:10.566135 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:10.567068 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.567569 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.568880 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.569177 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.569219 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.573882 (+ 4663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:10.573884 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:10.573884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:10.573893 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:10.578117 (+ 4224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.579202 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.579203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.579204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.579266 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.579476 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.579477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.579478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.579637 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:10.579715 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:10.579915 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 00:07:10.580013 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 00:07:10.580396 7455 heartbeater.cc:347] Connected to a master server at 127.0.98.61:33203
I20260620 00:07:10.580457 7455 heartbeater.cc:464] Registering TS with master...
I20260620 00:07:10.580536 7455 heartbeater.cc:511] Master 127.0.98.61:33203 requested a full tablet report, sending...
I20260620 00:07:10.580869 7087 ts_manager.cc:194] Registered new tserver with Master: ee53defd13044c57bbb138930c23dbf6 (127.0.98.1:44335)
I20260620 00:07:10.580935 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.565111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45835 (local address 127.0.98.1:35577)
0620 00:07:10.565256 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:10.565292 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:10.565953 (+ 661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:10.566204 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:10.566208 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:10.566767 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:10.566969 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.566974 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.567675 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.567675 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.568775 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.568777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.569296 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.569296 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.569395 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.570214 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:10.570219 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:10.573768 (+ 3549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:10.578213 (+ 4445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.578214 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.578215 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.579113 (+ 898us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.579341 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.579341 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.579342 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.579394 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.579744 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:10.579745 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:10.579791 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 00:07:10.580287 (+ 496us) client_negotiation.cc:240] Negotiation successful
0620 00:07:10.580409 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":3,"thread_start_us":85,"threads_started":1}
I20260620 00:07:10.581292 7453 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45835
I20260620 00:07:10.581346 7453 heartbeater.cc:464] Registering TS with master...
I20260620 00:07:10.581460 7453 heartbeater.cc:511] Master 127.0.98.62:45835 requested a full tablet report, sending...
I20260620 00:07:10.581732 6489 ts_manager.cc:194] Re-registered known tserver with Master: ee53defd13044c57bbb138930c23dbf6 (127.0.98.1:44335)
I20260620 00:07:10.582069 6489 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35577
W20260620 00:07:10.668725 7461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:10.668990 7461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:10.669034 7461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:10.670375 7461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:10.670455 7461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:10.670506 7461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:07:10.672104 7461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34161
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42847
--tserver_master_addrs=127.0.98.62:45835,127.0.98.61:33203,127.0.98.60:35537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:10.672405 7461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:10.672655 7461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:10.677971 7466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:10.677968 7469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:10.677981 7467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:10.678306 7461 server_base.cc:1034] running on GCE node
I20260620 00:07:10.678556 7461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:10.678809 7461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:10.679999 7461 hybrid_clock.cc:648] HybridClock initialized: now 1781914030679975 us; error 38 us; skew 500 ppm
Jun 20 00:07:10 dist-test-slave-grqt krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914030, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:10.683007 7461 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 00:07:10.683529 7461 webserver.cc:460] Webserver started at http://127.0.98.2:42847/ using document root <none> and password file <none>
I20260620 00:07:10.683794 7461 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:10.683856 7461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:10.685500 7461 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:07:10.686237 7476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:10.686532 7461 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:10.686637 7461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-1/wal
uuid: "4835482b92894a80985bac108abdac59"
format_stamp: "Formatted at 2026-06-20 00:07:09 on dist-test-slave-grqt"
server_key: "ca3b5c6986eb5adf7bef0f8ce2872390"
server_key_iv: "eedcda7d03fcc5be9d125b6a6fb84447"
server_key_version: "encryptionkey@0"
I20260620 00:07:10.687039 7461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:10.697314 7461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:10.699962 7461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:10.700133 7461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:10.700428 7461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:07:10.700759 7461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:07:10.700827 7461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:10.700879 7461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:07:10.700901 7461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:10.707377 7461 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34161
I20260620 00:07:10.707415 7589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34161 every 8 connection(s)
I20260620 00:07:10.707800 7461 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:07:10.714599 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 7461
I20260620 00:07:10.714795 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 6819
Jun 20 00:07:10 dist-test-slave-grqt krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914030, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 00:07:10 dist-test-slave-grqt krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914030, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 20 00:07:10 dist-test-slave-grqt krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914030, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 00:07:10.719791 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.709192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35697 (local address 127.0.98.60:35537)
0620 00:07:10.709238 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:10.709240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:10.709292 (+ 52us) server_negotiation.cc:407] Connection header received
0620 00:07:10.709793 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:10.709794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:10.709809 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:10.709841 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:10.710885 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.711648 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.712792 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.713102 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.713152 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.716560 (+ 3408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:10.716562 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:10.716562 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:10.716571 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:10.718460 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.718872 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.718873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.718873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.718931 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.719145 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.719146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.719146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.719279 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:10.719323 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:10.719596 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 00:07:10.719695 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 00:07:10.719825 7138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.708643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51489 (local address 127.0.98.61:33203)
0620 00:07:10.708718 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:10.708720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:10.709013 (+ 293us) server_negotiation.cc:407] Connection header received
0620 00:07:10.709792 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:10.709793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:10.709806 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:10.709841 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:10.710884 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.711654 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.712791 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.713086 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.713128 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.716055 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:10.716056 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:10.716056 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:10.716065 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:10.718097 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.718632 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.718634 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.718634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.718695 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.718994 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.718995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.718996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.719162 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:10.719205 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:10.719586 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 00:07:10.719675 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 00:07:10.720463 7594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.708645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33203 (local address 127.0.98.2:51489)
0620 00:07:10.708853 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:10.708894 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:10.709667 (+ 773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:10.709921 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:10.709926 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:10.710484 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:10.710771 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.710777 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.711751 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.711752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.712690 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.712691 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.713210 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.713211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.713276 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.713852 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:10.713856 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:10.715939 (+ 2083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:10.718200 (+ 2261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.718208 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.718211 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.718540 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.718768 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.718768 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.718769 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.718924 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.719250 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:10.719256 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:10.719437 (+ 181us) client_negotiation.cc:769] Sending connection context
0620 00:07:10.719948 (+ 511us) client_negotiation.cc:240] Negotiation successful
0620 00:07:10.720258 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":188,"thread_start_us":86,"threads_started":1}
I20260620 00:07:10.720468 7596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.709108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35537 (local address 127.0.98.2:35697)
0620 00:07:10.709236 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:10.709246 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:10.709676 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:10.709922 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:10.709926 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:10.710453 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:10.710771 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.710777 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.711751 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.711752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.712688 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.712690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.713221 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.713222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.713290 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.713938 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:10.713943 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:10.716480 (+ 2537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:10.718560 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.718562 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.718563 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.718781 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.719006 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.719007 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.719007 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.719070 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.719385 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:10.719386 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:10.719440 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 00:07:10.719902 (+ 462us) client_negotiation.cc:240] Negotiation successful
0620 00:07:10.720160 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":45,"thread_start_us":85,"threads_started":1}
I20260620 00:07:10.720667 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.709049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39873 (local address 127.0.98.62:45835)
0620 00:07:10.709248 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:10.709250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:10.709260 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:07:10.709771 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:10.709772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:10.709785 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:10.709831 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:10.710866 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.711395 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.712435 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.712656 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.712745 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.716900 (+ 4155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:10.716902 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:10.716902 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:10.716911 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:10.719033 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.719402 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.719404 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.719404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.719471 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.719676 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.719677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.719677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.719802 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:10.719858 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:10.720507 (+ 649us) server_negotiation.cc:299] Negotiation successful
0620 00:07:10.720593 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173}
I20260620 00:07:10.721179 7592 heartbeater.cc:347] Connected to a master server at 127.0.98.61:33203
I20260620 00:07:10.721285 7595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.708933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45835 (local address 127.0.98.2:39873)
0620 00:07:10.709052 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:10.709061 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:10.709667 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:10.710274 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:10.710275 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:10.710494 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:10.710771 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.710776 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.711513 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.711514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.712336 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.712338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.712757 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.712758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.712899 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.713475 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:10.713486 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:10.716801 (+ 3315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:10.719114 (+ 2313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.719115 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.719116 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.719316 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.719549 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.719550 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.719550 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.719606 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.720316 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:10.720317 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:10.720363 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 00:07:10.721029 (+ 666us) client_negotiation.cc:240] Negotiation successful
0620 00:07:10.721147 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":57,"thread_start_us":76,"threads_started":1}
I20260620 00:07:10.721303 7592 heartbeater.cc:464] Registering TS with master...
I20260620 00:07:10.721542 7592 heartbeater.cc:511] Master 127.0.98.61:33203 requested a full tablet report, sending...
I20260620 00:07:10.722116 7087 ts_manager.cc:194] Registered new tserver with Master: 4835482b92894a80985bac108abdac59 (127.0.98.2:34161)
I20260620 00:07:10.722298 7590 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45835
I20260620 00:07:10.722437 7591 heartbeater.cc:347] Connected to a master server at 127.0.98.60:35537
I20260620 00:07:10.722499 7590 heartbeater.cc:464] Registering TS with master...
I20260620 00:07:10.722530 7591 heartbeater.cc:464] Registering TS with master...
I20260620 00:07:10.722584 7590 heartbeater.cc:511] Master 127.0.98.62:45835 requested a full tablet report, sending...
I20260620 00:07:10.722608 7591 heartbeater.cc:511] Master 127.0.98.60:35537 requested a full tablet report, sending...
I20260620 00:07:10.722728 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46179
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45835
--webserver_port=41529
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:45835,127.0.98.61:33203
--tserver_master_addrs=127.0.98.62:45835,127.0.98.61:33203,127.0.98.60:35537 with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:07:10.722821 6489 ts_manager.cc:194] Re-registered known tserver with Master: 4835482b92894a80985bac108abdac59 (127.0.98.2:34161)
I20260620 00:07:10.722994 7274 ts_manager.cc:194] Registered new tserver with Master: 4835482b92894a80985bac108abdac59 (127.0.98.2:34161)
I20260620 00:07:10.723331 6489 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39873
W20260620 00:07:10.809245 7597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:10.809556 7597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:10.809587 7597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:10.810803 7597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:10.810848 7597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:10.810876 7597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:07:10.812341 7597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46179
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41529
--tserver_master_addrs=127.0.98.62:45835,127.0.98.61:33203,127.0.98.60:35537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:10.812546 7597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:10.812752 7597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:10.818339 7603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:10.818327 7602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:10.818451 7597 server_base.cc:1034] running on GCE node
W20260620 00:07:10.818332 7605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:10.818730 7597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:10.819020 7597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:10.820238 7597 hybrid_clock.cc:648] HybridClock initialized: now 1781914030820208 us; error 41 us; skew 500 ppm
Jun 20 00:07:10 dist-test-slave-grqt krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914030, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:10.823256 7597 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 00:07:10.823814 7597 webserver.cc:460] Webserver started at http://127.0.98.3:41529/ using document root <none> and password file <none>
I20260620 00:07:10.824079 7597 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:10.824173 7597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:10.825770 7597 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:07:10.826619 7612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:10.826936 7597 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:10.827047 7597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-2/wal
uuid: "87af16a18180461ca7afc0bff09802ab"
format_stamp: "Formatted at 2026-06-20 00:07:09 on dist-test-slave-grqt"
server_key: "f829a782da2388c379bbfad3fdfd23c3"
server_key_iv: "b0f73bea5106e65a66fcc4ec0c22820b"
server_key_version: "encryptionkey@0"
I20260620 00:07:10.827418 7597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:10.842046 7597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:10.845323 7597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:10.845536 7597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:10.845819 7597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:07:10.846175 7597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:07:10.846239 7597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:10.846293 7597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:07:10.846342 7597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:10.853142 7597 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46179
I20260620 00:07:10.853228 7725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46179 every 8 connection(s)
I20260620 00:07:10.853829 7597 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:07:10.858220 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 7597
Jun 20 00:07:10 dist-test-slave-grqt krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914030, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 00:07:10 dist-test-slave-grqt krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914030, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 20 00:07:10 dist-test-slave-grqt krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914030, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 00:07:10.871296 7138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.854911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38777 (local address 127.0.98.61:33203)
0620 00:07:10.855037 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:10.855040 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:10.855431 (+ 391us) server_negotiation.cc:407] Connection header received
0620 00:07:10.857257 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:10.857259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:10.857274 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:10.857357 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:10.858646 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.859387 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.861166 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.861512 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.861558 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.866932 (+ 5374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:10.866934 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:10.866934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:10.866943 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:10.868591 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.869641 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.869642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.869642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.869718 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.869921 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.869922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.869922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.870052 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:10.870104 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:10.871081 (+ 977us) server_negotiation.cc:299] Negotiation successful
0620 00:07:10.871173 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 00:07:10.871491 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.855322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38499 (local address 127.0.98.62:45835)
0620 00:07:10.855405 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:10.855408 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:10.856110 (+ 702us) server_negotiation.cc:407] Connection header received
0620 00:07:10.857258 (+ 1148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:10.857259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:10.857274 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:10.857357 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:10.858570 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.859370 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.861168 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.861514 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.861558 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.865634 (+ 4076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:10.865635 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:10.865636 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:10.865645 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:10.868672 (+ 3027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.869301 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.869302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.869303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.869391 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.869770 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.869771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.869772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.869926 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:10.870634 (+ 708us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:10.871298 (+ 664us) server_negotiation.cc:299] Negotiation successful
0620 00:07:10.871395 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 00:07:10.871979 7730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.854788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33203 (local address 127.0.98.3:38777)
0620 00:07:10.855035 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:10.855245 (+ 210us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:10.856944 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:10.857480 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:10.857485 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:10.858071 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:10.858489 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.858498 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.859532 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.859533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.861049 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.861052 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.861658 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.861659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.861842 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.862718 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:10.862731 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:10.866807 (+ 4076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:10.869287 (+ 2480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.869288 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.869289 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.869552 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.869797 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.869798 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.869798 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.869851 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.870141 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:10.870148 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:10.870504 (+ 356us) client_negotiation.cc:769] Sending connection context
0620 00:07:10.871459 (+ 955us) client_negotiation.cc:240] Negotiation successful
0620 00:07:10.871724 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":117,"thread_start_us":83,"threads_started":1}
I20260620 00:07:10.872444 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.855126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45835 (local address 127.0.98.3:38499)
0620 00:07:10.855502 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:10.855515 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:10.856945 (+ 1430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:10.857623 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:10.857624 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:10.858057 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:10.858429 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.858437 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.859532 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.859533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.861049 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.861052 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.861659 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.861660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.861842 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.862762 (+ 920us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:10.862765 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:10.865510 (+ 2745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:10.868795 (+ 3285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.868809 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.868812 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.869183 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.869468 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.869469 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.869469 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.869687 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.870692 (+ 1005us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:10.870693 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:10.870752 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 00:07:10.871995 (+ 1243us) client_negotiation.cc:240] Negotiation successful
0620 00:07:10.872330 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":202,"thread_start_us":63,"threads_started":1}
I20260620 00:07:10.873057 7726 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45835
I20260620 00:07:10.873178 7726 heartbeater.cc:464] Registering TS with master...
I20260620 00:07:10.873447 7726 heartbeater.cc:511] Master 127.0.98.62:45835 requested a full tablet report, sending...
I20260620 00:07:10.873525 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.855479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54953 (local address 127.0.98.60:35537)
0620 00:07:10.855539 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:10.855541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:10.857252 (+ 1711us) server_negotiation.cc:407] Connection header received
0620 00:07:10.857316 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:10.857317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:10.857330 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:10.857402 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:10.858570 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.859363 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.861165 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.861518 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.861561 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.867303 (+ 5742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:10.867305 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:10.867305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:10.867314 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:10.869192 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.872457 (+ 3265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.872459 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.872459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.872533 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.872867 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.872868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.872869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.873023 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:10.873075 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:10.873306 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 00:07:10.873417 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 00:07:10.873706 7728 heartbeater.cc:347] Connected to a master server at 127.0.98.61:33203
I20260620 00:07:10.873754 7728 heartbeater.cc:464] Registering TS with master...
I20260620 00:07:10.873828 7728 heartbeater.cc:511] Master 127.0.98.61:33203 requested a full tablet report, sending...
I20260620 00:07:10.873924 6489 ts_manager.cc:194] Re-registered known tserver with Master: 87af16a18180461ca7afc0bff09802ab (127.0.98.3:46179)
I20260620 00:07:10.874109 7087 ts_manager.cc:194] Registered new tserver with Master: 87af16a18180461ca7afc0bff09802ab (127.0.98.3:46179)
I20260620 00:07:10.874341 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.855292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35537 (local address 127.0.98.3:54953)
0620 00:07:10.857165 (+ 1873us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:10.857175 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:10.857253 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:10.857480 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:10.857485 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:10.858127 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:10.858429 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.858437 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.859532 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.859533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.861054 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.861057 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.861661 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.861662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.861843 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.863257 (+ 1414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:10.863263 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:10.866275 (+ 3012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:10.870651 (+ 4376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.870652 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.870652 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.870951 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.872618 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:10.872619 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:10.872619 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:10.872674 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:10.873116 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:10.873117 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:10.873204 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 00:07:10.873277 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 00:07:10.873419 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1854,"mutex_wait_us":78,"thread_start_us":62,"threads_started":1}
I20260620 00:07:10.874493 6489 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38499
I20260620 00:07:10.874727 7727 heartbeater.cc:347] Connected to a master server at 127.0.98.60:35537
I20260620 00:07:10.874789 7727 heartbeater.cc:464] Registering TS with master...
I20260620 00:07:10.874864 7727 heartbeater.cc:511] Master 127.0.98.60:35537 requested a full tablet report, sending...
I20260620 00:07:10.875185 7274 ts_manager.cc:194] Registered new tserver with Master: 87af16a18180461ca7afc0bff09802ab (127.0.98.3:46179)
I20260620 00:07:10.875993 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.860804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45792 (local address 127.0.98.60:35537)
0620 00:07:10.861068 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:10.861069 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:10.862409 (+ 1340us) server_negotiation.cc:407] Connection header received
0620 00:07:10.862439 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:10.862440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:10.862455 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:10.862563 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:10.864790 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.865590 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.866603 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.866926 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.866986 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.869588 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:10.869590 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:10.869590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:10.869597 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:10.872659 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.874703 (+ 2044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.874704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.874704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.874770 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.874997 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.874998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.874998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.875464 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:10.875574 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:10.875706 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 00:07:10.875829 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":52,"threads_started":1}
I20260620 00:07:10.877562 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.860427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34198 (local address 127.0.98.61:33203)
0620 00:07:10.861017 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:10.861019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:10.861299 (+ 280us) server_negotiation.cc:407] Connection header received
0620 00:07:10.861423 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:10.861424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:10.861439 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:10.861478 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:10.867342 (+ 5864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.868188 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.870264 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.870705 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.870792 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.872146 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:10.872148 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:10.872148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:10.872156 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:10.874135 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.876555 (+ 2420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.876557 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.876557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.876629 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.876828 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.876829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.876829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.876948 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:10.877066 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:10.877195 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 00:07:10.877328 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":41,"threads_started":1}
I20260620 00:07:10.878103 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.860011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39252 (local address 127.0.98.62:45835)
0620 00:07:10.862394 (+ 2383us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:10.862396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:10.862403 (+ 7us) server_negotiation.cc:407] Connection header received
0620 00:07:10.862426 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:10.862427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:10.862442 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:10.865140 (+ 2698us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:10.865189 (+ 49us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.865923 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.869321 (+ 3398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.869686 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.869851 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.872784 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:10.872786 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:10.872786 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:10.872794 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:10.876688 (+ 3894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.877034 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.877036 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.877036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.877097 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:10.877324 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:10.877325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:10.877325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:10.877580 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:10.877791 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:10.877867 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 00:07:10.878037 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2351,"thread_start_us":83,"threads_started":1}
I20260620 00:07:10.880707 6489 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39252:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:07:10.882539 6489 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:07:10.892520 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.887681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46179 (local address 127.0.0.1:46132)
0620 00:07:10.887937 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:10.887947 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:10.887992 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:10.888342 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:10.888343 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:10.888346 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:07:10.888414 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.888417 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.889999 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.890000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.891309 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.891313 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.892258 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.892259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.892328 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.892334 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 00:07:10.892369 (+ 35us) client_negotiation.cc:240] Negotiation successful
0620 00:07:10.892424 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":98,"thread_start_us":100,"threads_started":1}
I20260620 00:07:10.892776 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.887678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44335 (local address 127.0.0.1:58416)
0620 00:07:10.887788 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:10.887803 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:10.887854 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:10.888917 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:10.888918 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:10.888920 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:07:10.888955 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.888956 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.890381 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.890382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.891531 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.891533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.892488 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.892489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.892562 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.892566 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:07:10.892620 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 00:07:10.892697 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 00:07:10.892788 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.888158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58416 (local address 127.0.98.1:44335)
0620 00:07:10.888367 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:10.888372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:10.888384 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:07:10.888658 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:10.888679 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:10.888736 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:10.888902 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:07:10.889058 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.890289 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.891627 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.892384 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.892462 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.892545 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:10.892635 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 00:07:10.892701 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":101,"threads_started":1}
I20260620 00:07:10.892964 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.887913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34161 (local address 127.0.0.1:50082)
0620 00:07:10.888614 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:10.888624 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:10.888668 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:10.889618 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:10.889619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:10.889622 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:07:10.889670 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.889672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.890867 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.890867 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.891858 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:10.891860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.892717 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.892718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:10.892779 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.892783 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:07:10.892835 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 00:07:10.892882 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":92,"threads_started":1}
I20260620 00:07:10.892993 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.887948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46132 (local address 127.0.98.3:46179)
0620 00:07:10.888120 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:10.888126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:10.888137 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:07:10.888183 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:10.888186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:10.888205 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:10.888493 (+ 288us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:07:10.888555 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.889858 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.891384 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.892130 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.892752 (+ 622us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.892846 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:10.892874 (+ 28us) server_negotiation.cc:299] Negotiation successful
0620 00:07:10.892921 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":109,"threads_started":1}
I20260620 00:07:10.893226 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:10.888148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50082 (local address 127.0.98.2:34161)
0620 00:07:10.889095 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:10.889100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:10.889112 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:07:10.889156 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:10.889158 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:10.889172 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:10.889649 (+ 477us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:07:10.889742 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.890773 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.891958 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:10.892518 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:10.892596 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:10.892682 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:10.893060 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 00:07:10.893120 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":888,"thread_start_us":144,"threads_started":1}
I20260620 00:07:10.893910 7387 tablet_service.cc:1467] Processing CreateTablet for tablet a19f1051d0b64a5484e4dcc972b1a360 (DEFAULT_TABLE table=test-table [id=8229e4f6d3f74a50925ce2f1204f5459]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:07:10.893877 7660 tablet_service.cc:1467] Processing CreateTablet for tablet a19f1051d0b64a5484e4dcc972b1a360 (DEFAULT_TABLE table=test-table [id=8229e4f6d3f74a50925ce2f1204f5459]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:07:10.894101 7524 tablet_service.cc:1467] Processing CreateTablet for tablet a19f1051d0b64a5484e4dcc972b1a360 (DEFAULT_TABLE table=test-table [id=8229e4f6d3f74a50925ce2f1204f5459]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:07:10.894294 7660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a19f1051d0b64a5484e4dcc972b1a360. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:10.894294 7387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a19f1051d0b64a5484e4dcc972b1a360. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:10.894387 7524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a19f1051d0b64a5484e4dcc972b1a360. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:10.898384 7754 tablet_bootstrap.cc:492] T a19f1051d0b64a5484e4dcc972b1a360 P 4835482b92894a80985bac108abdac59: Bootstrap starting.
I20260620 00:07:10.899042 7754 tablet_bootstrap.cc:654] T a19f1051d0b64a5484e4dcc972b1a360 P 4835482b92894a80985bac108abdac59: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:10.899397 7754 log.cc:824] T a19f1051d0b64a5484e4dcc972b1a360 P 4835482b92894a80985bac108abdac59: Log is configured to *not* fsync() on all Append() calls
I20260620 00:07:10.899828 7752 tablet_bootstrap.cc:492] T a19f1051d0b64a5484e4dcc972b1a360 P 87af16a18180461ca7afc0bff09802ab: Bootstrap starting.
I20260620 00:07:10.900453 7754 tablet_bootstrap.cc:492] T a19f1051d0b64a5484e4dcc972b1a360 P 4835482b92894a80985bac108abdac59: No bootstrap required, opened a new log
I20260620 00:07:10.900545 7754 ts_tablet_manager.cc:1397] T a19f1051d0b64a5484e4dcc972b1a360 P 4835482b92894a80985bac108abdac59: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:07:10.900573 7752 tablet_bootstrap.cc:654] T a19f1051d0b64a5484e4dcc972b1a360 P 87af16a18180461ca7afc0bff09802ab: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:10.901417 7752 log.cc:824] T a19f1051d0b64a5484e4dcc972b1a360 P 87af16a18180461ca7afc0bff09802ab: Log is configured to *not* fsync() on all Append() calls
I20260620 00:07:10.901633 7753 tablet_bootstrap.cc:492] T a19f1051d0b64a5484e4dcc972b1a360 P ee53defd13044c57bbb138930c23dbf6: Bootstrap starting.
I20260620 00:07:10.902279 7753 tablet_bootstrap.cc:654] T a19f1051d0b64a5484e4dcc972b1a360 P ee53defd13044c57bbb138930c23dbf6: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:10.902418 7752 tablet_bootstrap.cc:492] T a19f1051d0b64a5484e4dcc972b1a360 P 87af16a18180461ca7afc0bff09802ab: No bootstrap required, opened a new log
I20260620 00:07:10.902529 7752 ts_tablet_manager.cc:1397] T a19f1051d0b64a5484e4dcc972b1a360 P 87af16a18180461ca7afc0bff09802ab: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 00:07:10.902549 7753 log.cc:824] T a19f1051d0b64a5484e4dcc972b1a360 P ee53defd13044c57bbb138930c23dbf6: Log is configured to *not* fsync() on all Append() calls
I20260620 00:07:10.903244 7753 tablet_bootstrap.cc:492] T a19f1051d0b64a5484e4dcc972b1a360 P ee53defd13044c57bbb138930c23dbf6: No bootstrap required, opened a new log
I20260620 00:07:10.903307 7753 ts_tablet_manager.cc:1397] T a19f1051d0b64a5484e4dcc972b1a360 P ee53defd13044c57bbb138930c23dbf6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:07:10.903214 7754 raft_consensus.cc:348] T a19f1051d0b64a5484e4dcc972b1a360 P 4835482b92894a80985bac108abdac59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee53defd13044c57bbb138930c23dbf6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44335 } } peers { permanent_uuid: "87af16a18180461ca7afc0bff09802ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46179 } } peers { permanent_uuid: "4835482b92894a80985bac108abdac59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34161 } }
I20260620 00:07:10.903409 7754 raft_consensus.cc:374] T a19f1051d0b64a5484e4dcc972b1a360 P 4835482b92894a80985bac108abdac59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:10.903452 7754 raft_consensus.cc:729] T a19f1051d0b64a5484e4dcc972b1a360 P 4835482b92894a80985bac108abdac59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4835482b92894a80985bac108abdac59, State: Initialized, Role: FOLLOWER
I20260620 00:07:10.903580 7754 consensus_queue.cc:260] T a19f1051d0b64a5484e4dcc972b1a360 P 4835482b92894a80985bac108abdac59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee53defd13044c57bbb138930c23dbf6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44335 } } peers { permanent_uuid: "87af16a18180461ca7afc0bff09802ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46179 } } peers { permanent_uuid: "4835482b92894a80985bac108abdac59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34161 } }
I20260620 00:07:10.903898 7754 ts_tablet_manager.cc:1428] T a19f1051d0b64a5484e4dcc972b1a360 P 4835482b92894a80985bac108abdac59: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260620 00:07:10.903952 7590 heartbeater.cc:503] Master 127.0.98.62:45835 was elected leader, sending a full tablet report...
I20260620 00:07:10.906104 7752 raft_consensus.cc:348] T a19f1051d0b64a5484e4dcc972b1a360 P 87af16a18180461ca7afc0bff09802ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee53defd13044c57bbb138930c23dbf6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44335 } } peers { permanent_uuid: "87af16a18180461ca7afc0bff09802ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46179 } } peers { permanent_uuid: "4835482b92894a80985bac108abdac59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34161 } }
I20260620 00:07:10.906289 7752 raft_consensus.cc:374] T a19f1051d0b64a5484e4dcc972b1a360 P 87af16a18180461ca7afc0bff09802ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:10.906337 7752 raft_consensus.cc:729] T a19f1051d0b64a5484e4dcc972b1a360 P 87af16a18180461ca7afc0bff09802ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87af16a18180461ca7afc0bff09802ab, State: Initialized, Role: FOLLOWER
I20260620 00:07:10.906464 7752 consensus_queue.cc:260] T a19f1051d0b64a5484e4dcc972b1a360 P 87af16a18180461ca7afc0bff09802ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee53defd13044c57bbb138930c23dbf6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44335 } } peers { permanent_uuid: "87af16a18180461ca7afc0bff09802ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46179 } } peers { permanent_uuid: "4835482b92894a80985bac108abdac59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34161 } }
I20260620 00:07:10.906744 7726 heartbeater.cc:503] Master 127.0.98.62:45835 was elected leader, sending a full tablet report...
I20260620 00:07:10.906695 7753 raft_consensus.cc:348] T a19f1051d0b64a5484e4dcc972b1a360 P ee53defd13044c57bbb138930c23dbf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee53defd13044c57bbb138930c23dbf6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44335 } } peers { permanent_uuid: "87af16a18180461ca7afc0bff09802ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46179 } } peers { permanent_uuid: "4835482b92894a80985bac108abdac59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34161 } }
I20260620 00:07:10.906875 7753 raft_consensus.cc:374] T a19f1051d0b64a5484e4dcc972b1a360 P ee53defd13044c57bbb138930c23dbf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:10.906919 7753 raft_consensus.cc:729] T a19f1051d0b64a5484e4dcc972b1a360 P ee53defd13044c57bbb138930c23dbf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee53defd13044c57bbb138930c23dbf6, State: Initialized, Role: FOLLOWER
I20260620 00:07:10.907066 7753 consensus_queue.cc:260] T a19f1051d0b64a5484e4dcc972b1a360 P ee53defd13044c57bbb138930c23dbf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee53defd13044c57bbb138930c23dbf6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44335 } } peers { permanent_uuid: "87af16a18180461ca7afc0bff09802ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46179 } } peers { permanent_uuid: "4835482b92894a80985bac108abdac59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34161 } }
I20260620 00:07:10.907301 7752 ts_tablet_manager.cc:1428] T a19f1051d0b64a5484e4dcc972b1a360 P 87af16a18180461ca7afc0bff09802ab: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260620 00:07:10.907338 7753 ts_tablet_manager.cc:1428] T a19f1051d0b64a5484e4dcc972b1a360 P ee53defd13044c57bbb138930c23dbf6: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 00:07:10.907481 7453 heartbeater.cc:503] Master 127.0.98.62:45835 was elected leader, sending a full tablet report...
W20260620 00:07:10.958609 7593 tablet.cc:2369] T a19f1051d0b64a5484e4dcc972b1a360 P 4835482b92894a80985bac108abdac59: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:07:11.021306 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:11.018804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45806 (local address 127.0.98.60:35537)
0620 00:07:11.018865 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:11.018867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:11.018937 (+ 70us) server_negotiation.cc:407] Connection header received
0620 00:07:11.018988 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:11.018989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:11.019010 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:11.019087 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:07:11.019204 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:11.019720 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:11.020614 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:11.021013 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:11.021076 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:11.021135 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:11.021213 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 00:07:11.021246 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 00:07:11.021303 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:11.018685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35537 (local address 127.0.0.1:45806)
0620 00:07:11.018877 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:11.018888 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:11.018939 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:11.019099 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:11.019100 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:11.019101 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:07:11.019149 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:11.019151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:11.019790 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:11.019790 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:11.020529 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:11.020531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:11.021094 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:11.021095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:11.021146 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:11.021149 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:07:11.021196 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 00:07:11.021237 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260620 00:07:11.021840 7284 raft_consensus.cc:1203] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260620 00:07:11.022743 7761 sys_catalog.cc:455] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60dfbd8a9e3a4e4aaad18e7dac6d0ea1. Latest consensus state: current_term: 1 leader_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "b410653b3fa54e709136276f3e32a9bb" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35537 } attrs { promote: true } } }
I20260620 00:07:11.022890 7761 sys_catalog.cc:458] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [sys.catalog]: This master's current role is: LEARNER
I20260620 00:07:11.042454 7759 raft_consensus.cc:482] T a19f1051d0b64a5484e4dcc972b1a360 P 87af16a18180461ca7afc0bff09802ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:07:11.042580 7759 raft_consensus.cc:504] T a19f1051d0b64a5484e4dcc972b1a360 P 87af16a18180461ca7afc0bff09802ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee53defd13044c57bbb138930c23dbf6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44335 } } peers { permanent_uuid: "87af16a18180461ca7afc0bff09802ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46179 } } peers { permanent_uuid: "4835482b92894a80985bac108abdac59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34161 } }
I20260620 00:07:11.042876 7759 leader_election.cc:288] T a19f1051d0b64a5484e4dcc972b1a360 P 87af16a18180461ca7afc0bff09802ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee53defd13044c57bbb138930c23dbf6 (127.0.98.1:44335), 4835482b92894a80985bac108abdac59 (127.0.98.2:34161)
I20260620 00:07:11.047273 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:11.043212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52555 (local address 127.0.98.1:44335)
0620 00:07:11.043489 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:11.043490 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:11.043498 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:07:11.043523 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:11.043524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:11.043536 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:11.043608 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:07:11.043777 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:11.044557 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:11.046209 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:11.046797 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:11.046889 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:11.046901 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:11.047150 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 00:07:11.047203 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251}
I20260620 00:07:11.047266 7730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:11.043227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34161 (local address 127.0.98.3:51791)
0620 00:07:11.043271 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:11.043284 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:11.043332 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:11.043542 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:11.043543 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:11.043545 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:07:11.043657 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:11.043660 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:11.044671 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:11.044672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:11.046072 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:11.046075 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:11.046924 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:11.046925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:11.047016 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:11.047023 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 00:07:11.047098 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 00:07:11.047154 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":86}
I20260620 00:07:11.047313 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:11.043336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51791 (local address 127.0.98.2:34161)
0620 00:07:11.043380 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:11.043382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:11.043395 (+ 13us) server_negotiation.cc:407] Connection header received
0620 00:07:11.043436 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:11.043437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:11.043449 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:11.043504 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:07:11.043837 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:11.044588 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:11.046209 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:11.046813 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:11.046889 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:11.046901 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:11.047150 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 00:07:11.047244 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 00:07:11.047379 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:11.043107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44335 (local address 127.0.98.3:52555)
0620 00:07:11.043234 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:11.043250 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:11.043342 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:11.043628 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:11.043629 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:11.043631 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:07:11.043696 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:11.043698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:11.044671 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:11.044672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:11.046072 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:11.046075 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:11.046924 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:11.046925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:11.047016 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:11.047023 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 00:07:11.047098 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 00:07:11.047154 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":49}
I20260620 00:07:11.047811 7544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19f1051d0b64a5484e4dcc972b1a360" candidate_uuid: "87af16a18180461ca7afc0bff09802ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4835482b92894a80985bac108abdac59" is_pre_election: true
I20260620 00:07:11.047812 7407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19f1051d0b64a5484e4dcc972b1a360" candidate_uuid: "87af16a18180461ca7afc0bff09802ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee53defd13044c57bbb138930c23dbf6" is_pre_election: true
I20260620 00:07:11.047978 7407 raft_consensus.cc:2454] T a19f1051d0b64a5484e4dcc972b1a360 P ee53defd13044c57bbb138930c23dbf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87af16a18180461ca7afc0bff09802ab in term 0.
I20260620 00:07:11.047978 7544 raft_consensus.cc:2454] T a19f1051d0b64a5484e4dcc972b1a360 P 4835482b92894a80985bac108abdac59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87af16a18180461ca7afc0bff09802ab in term 0.
I20260620 00:07:11.048231 7616 leader_election.cc:302] T a19f1051d0b64a5484e4dcc972b1a360 P 87af16a18180461ca7afc0bff09802ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4835482b92894a80985bac108abdac59, 87af16a18180461ca7afc0bff09802ab; no voters:
I20260620 00:07:11.048404 7759 raft_consensus.cc:2781] T a19f1051d0b64a5484e4dcc972b1a360 P 87af16a18180461ca7afc0bff09802ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:07:11.048484 7759 raft_consensus.cc:482] T a19f1051d0b64a5484e4dcc972b1a360 P 87af16a18180461ca7afc0bff09802ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:07:11.048520 7759 raft_consensus.cc:3037] T a19f1051d0b64a5484e4dcc972b1a360 P 87af16a18180461ca7afc0bff09802ab [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:11.049258 7759 raft_consensus.cc:504] T a19f1051d0b64a5484e4dcc972b1a360 P 87af16a18180461ca7afc0bff09802ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee53defd13044c57bbb138930c23dbf6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44335 } } peers { permanent_uuid: "87af16a18180461ca7afc0bff09802ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46179 } } peers { permanent_uuid: "4835482b92894a80985bac108abdac59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34161 } }
I20260620 00:07:11.049444 7759 leader_election.cc:288] T a19f1051d0b64a5484e4dcc972b1a360 P 87af16a18180461ca7afc0bff09802ab [CANDIDATE]: Term 1 election: Requested vote from peers ee53defd13044c57bbb138930c23dbf6 (127.0.98.1:44335), 4835482b92894a80985bac108abdac59 (127.0.98.2:34161)
I20260620 00:07:11.049643 7544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19f1051d0b64a5484e4dcc972b1a360" candidate_uuid: "87af16a18180461ca7afc0bff09802ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4835482b92894a80985bac108abdac59"
I20260620 00:07:11.049643 7407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19f1051d0b64a5484e4dcc972b1a360" candidate_uuid: "87af16a18180461ca7afc0bff09802ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee53defd13044c57bbb138930c23dbf6"
I20260620 00:07:11.049739 7544 raft_consensus.cc:3037] T a19f1051d0b64a5484e4dcc972b1a360 P 4835482b92894a80985bac108abdac59 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:11.049739 7407 raft_consensus.cc:3037] T a19f1051d0b64a5484e4dcc972b1a360 P ee53defd13044c57bbb138930c23dbf6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:11.050768 7407 raft_consensus.cc:2454] T a19f1051d0b64a5484e4dcc972b1a360 P ee53defd13044c57bbb138930c23dbf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87af16a18180461ca7afc0bff09802ab in term 1.
I20260620 00:07:11.050974 7616 leader_election.cc:302] T a19f1051d0b64a5484e4dcc972b1a360 P 87af16a18180461ca7afc0bff09802ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87af16a18180461ca7afc0bff09802ab, ee53defd13044c57bbb138930c23dbf6; no voters:
I20260620 00:07:11.051120 7759 raft_consensus.cc:2781] T a19f1051d0b64a5484e4dcc972b1a360 P 87af16a18180461ca7afc0bff09802ab [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:07:11.051162 7544 raft_consensus.cc:2454] T a19f1051d0b64a5484e4dcc972b1a360 P 4835482b92894a80985bac108abdac59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87af16a18180461ca7afc0bff09802ab in term 1.
I20260620 00:07:11.051312 7759 raft_consensus.cc:686] T a19f1051d0b64a5484e4dcc972b1a360 P 87af16a18180461ca7afc0bff09802ab [term 1 LEADER]: Becoming Leader. State: Replica: 87af16a18180461ca7afc0bff09802ab, State: Running, Role: LEADER
I20260620 00:07:11.051419 7759 consensus_queue.cc:237] T a19f1051d0b64a5484e4dcc972b1a360 P 87af16a18180461ca7afc0bff09802ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee53defd13044c57bbb138930c23dbf6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44335 } } peers { permanent_uuid: "87af16a18180461ca7afc0bff09802ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46179 } } peers { permanent_uuid: "4835482b92894a80985bac108abdac59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34161 } }
I20260620 00:07:11.052258 6488 catalog_manager.cc:5495] T a19f1051d0b64a5484e4dcc972b1a360 P 87af16a18180461ca7afc0bff09802ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 87af16a18180461ca7afc0bff09802ab (127.0.98.3). New cstate: current_term: 1 leader_uuid: "87af16a18180461ca7afc0bff09802ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee53defd13044c57bbb138930c23dbf6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87af16a18180461ca7afc0bff09802ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4835482b92894a80985bac108abdac59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34161 } health_report { overall_health: UNKNOWN } } }
I20260620 00:07:11.052848 7049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "b410653b3fa54e709136276f3e32a9bb" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35537 } attrs { promote: true } } }
I20260620 00:07:11.052953 7049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [sys.catalog]: This master's current role is: LEADER
I20260620 00:07:11.053632 7767 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1: attempting to promote NON_VOTER b410653b3fa54e709136276f3e32a9bb to VOTER
I20260620 00:07:11.053807 7767 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "b410653b3fa54e709136276f3e32a9bb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35537 } attrs { promote: false } }
I20260620 00:07:11.054370 7284 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [term 1 LEARNER]: Refusing update from remote peer 60dfbd8a9e3a4e4aaad18e7dac6d0ea1: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260620 00:07:11.054363 7097 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [term 1 FOLLOWER]: Refusing update from remote peer 60dfbd8a9e3a4e4aaad18e7dac6d0ea1: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260620 00:07:11.054704 7049 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260620 00:07:11.054853 7767 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b410653b3fa54e709136276f3e32a9bb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35537 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260620 00:07:11.055379 7049 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, b410653b3fa54e709136276f3e32a9bb (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "b410653b3fa54e709136276f3e32a9bb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35537 } attrs { promote: false } } }
I20260620 00:07:11.055953 7284 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, b410653b3fa54e709136276f3e32a9bb (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "b410653b3fa54e709136276f3e32a9bb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35537 } attrs { promote: false } } }
I20260620 00:07:11.056546 7049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "b410653b3fa54e709136276f3e32a9bb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35537 } attrs { promote: false } } }
I20260620 00:07:11.056629 7049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [sys.catalog]: This master's current role is: LEADER
I20260620 00:07:11.056595 7767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "b410653b3fa54e709136276f3e32a9bb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35537 } attrs { promote: false } } }
I20260620 00:07:11.056689 7767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [sys.catalog]: This master's current role is: LEADER
I20260620 00:07:11.056676 7049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "b410653b3fa54e709136276f3e32a9bb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35537 } attrs { promote: false } } }
I20260620 00:07:11.056723 7049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 [sys.catalog]: This master's current role is: LEADER
I20260620 00:07:11.056810 7097 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, b410653b3fa54e709136276f3e32a9bb (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "b410653b3fa54e709136276f3e32a9bb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35537 } attrs { promote: false } } }
I20260620 00:07:11.057111 7761 sys_catalog.cc:455] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "b410653b3fa54e709136276f3e32a9bb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35537 } attrs { promote: false } } }
I20260620 00:07:11.057191 7761 sys_catalog.cc:458] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:07:11.057823 7769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "b410653b3fa54e709136276f3e32a9bb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35537 } attrs { promote: false } } }
I20260620 00:07:11.057925 7769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 00:07:11.065438 7456 tablet.cc:2369] T a19f1051d0b64a5484e4dcc972b1a360 P ee53defd13044c57bbb138930c23dbf6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:07:11.082268 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:11.079090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46138 (local address 127.0.98.3:46179)
0620 00:07:11.079191 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:11.079194 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:11.079201 (+ 7us) server_negotiation.cc:407] Connection header received
0620 00:07:11.079237 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:11.079237 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:11.079258 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:11.079448 (+ 190us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:07:11.079607 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:11.080209 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:11.081055 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:11.081297 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:11.081361 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:11.081664 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:07:11.081818 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:07:11.081900 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:11.082134 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 00:07:11.082164 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 00:07:11.084529 7407 raft_consensus.cc:1261] T a19f1051d0b64a5484e4dcc972b1a360 P ee53defd13044c57bbb138930c23dbf6 [term 1 FOLLOWER]: Refusing update from remote peer 87af16a18180461ca7afc0bff09802ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:07:11.084545 7544 raft_consensus.cc:1261] T a19f1051d0b64a5484e4dcc972b1a360 P 4835482b92894a80985bac108abdac59 [term 1 FOLLOWER]: Refusing update from remote peer 87af16a18180461ca7afc0bff09802ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:07:11.084954 7765 consensus_queue.cc:1035] T a19f1051d0b64a5484e4dcc972b1a360 P 87af16a18180461ca7afc0bff09802ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee53defd13044c57bbb138930c23dbf6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:07:11.085194 7759 consensus_queue.cc:1035] T a19f1051d0b64a5484e4dcc972b1a360 P 87af16a18180461ca7afc0bff09802ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "4835482b92894a80985bac108abdac59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:07:11.088037 7771 mvcc.cc:205] Tried to move back new op lower bound from 7298719871318732800 to 7298719871187238912. Current Snapshot: MvccSnapshot[applied={T|T < 7298719871318732800}]
I20260620 00:07:11.091152 6488 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:07:11.091276 6488 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:07:11.093196 6488 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 60dfbd8a9e3a4e4aaad18e7dac6d0ea1: Sending DeleteTablet for 3 replicas of tablet a19f1051d0b64a5484e4dcc972b1a360
I20260620 00:07:11.093621 7387 tablet_service.cc:1514] Processing DeleteTablet for tablet a19f1051d0b64a5484e4dcc972b1a360 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:07:11 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58416
I20260620 00:07:11.093624 7524 tablet_service.cc:1514] Processing DeleteTablet for tablet a19f1051d0b64a5484e4dcc972b1a360 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:07:11 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50082
I20260620 00:07:11.093967 7660 tablet_service.cc:1514] Processing DeleteTablet for tablet a19f1051d0b64a5484e4dcc972b1a360 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:07:11 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46132
I20260620 00:07:11.094166 7781 tablet_replica.cc:330] T a19f1051d0b64a5484e4dcc972b1a360 P 4835482b92894a80985bac108abdac59: stopping tablet replica
I20260620 00:07:11.094293 7781 raft_consensus.cc:2229] T a19f1051d0b64a5484e4dcc972b1a360 P 4835482b92894a80985bac108abdac59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:07:11.094437 7781 raft_consensus.cc:2258] T a19f1051d0b64a5484e4dcc972b1a360 P 4835482b92894a80985bac108abdac59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:07:11.094600 7780 tablet_replica.cc:330] T a19f1051d0b64a5484e4dcc972b1a360 P ee53defd13044c57bbb138930c23dbf6: stopping tablet replica
I20260620 00:07:11.094746 7780 raft_consensus.cc:2229] T a19f1051d0b64a5484e4dcc972b1a360 P ee53defd13044c57bbb138930c23dbf6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:07:11.094861 7780 raft_consensus.cc:2258] T a19f1051d0b64a5484e4dcc972b1a360 P ee53defd13044c57bbb138930c23dbf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:07:11.095263 7782 tablet_replica.cc:330] T a19f1051d0b64a5484e4dcc972b1a360 P 87af16a18180461ca7afc0bff09802ab: stopping tablet replica
I20260620 00:07:11.095327 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 6457
I20260620 00:07:11.096045 7781 ts_tablet_manager.cc:1905] T a19f1051d0b64a5484e4dcc972b1a360 P 4835482b92894a80985bac108abdac59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:07:11.096228 7780 ts_tablet_manager.cc:1905] T a19f1051d0b64a5484e4dcc972b1a360 P ee53defd13044c57bbb138930c23dbf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:07:11.097991 7782 raft_consensus.cc:2229] T a19f1051d0b64a5484e4dcc972b1a360 P 87af16a18180461ca7afc0bff09802ab [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:07:11.098151 7782 raft_consensus.cc:2258] T a19f1051d0b64a5484e4dcc972b1a360 P 87af16a18180461ca7afc0bff09802ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:07:11.098320 7780 ts_tablet_manager.cc:1918] T a19f1051d0b64a5484e4dcc972b1a360 P ee53defd13044c57bbb138930c23dbf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:07:11.098460 7780 log.cc:1196] T a19f1051d0b64a5484e4dcc972b1a360 P ee53defd13044c57bbb138930c23dbf6: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-0/wal/wals/a19f1051d0b64a5484e4dcc972b1a360
I20260620 00:07:11.098718 7782 ts_tablet_manager.cc:1905] T a19f1051d0b64a5484e4dcc972b1a360 P 87af16a18180461ca7afc0bff09802ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:07:11.098949 7780 ts_tablet_manager.cc:1939] T a19f1051d0b64a5484e4dcc972b1a360 P ee53defd13044c57bbb138930c23dbf6: Deleting consensus metadata
I20260620 00:07:11.099277 7781 ts_tablet_manager.cc:1918] T a19f1051d0b64a5484e4dcc972b1a360 P 4835482b92894a80985bac108abdac59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:07:11.099380 7781 log.cc:1196] T a19f1051d0b64a5484e4dcc972b1a360 P 4835482b92894a80985bac108abdac59: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-1/wal/wals/a19f1051d0b64a5484e4dcc972b1a360
I20260620 00:07:11.099689 7781 ts_tablet_manager.cc:1939] T a19f1051d0b64a5484e4dcc972b1a360 P 4835482b92894a80985bac108abdac59: Deleting consensus metadata
I20260620 00:07:11.100880 7782 ts_tablet_manager.cc:1918] T a19f1051d0b64a5484e4dcc972b1a360 P 87af16a18180461ca7afc0bff09802ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:07:11.100986 7782 log.cc:1196] T a19f1051d0b64a5484e4dcc972b1a360 P 87af16a18180461ca7afc0bff09802ab: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-2/wal/wals/a19f1051d0b64a5484e4dcc972b1a360
I20260620 00:07:11.101257 7782 ts_tablet_manager.cc:1939] T a19f1051d0b64a5484e4dcc972b1a360 P 87af16a18180461ca7afc0bff09802ab: Deleting consensus metadata
W20260620 00:07:11.102124 7478 connection.cc:535] server connection from 127.0.0.1:50082 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 00:07:11.102150 7614 connection.cc:535] server connection from 127.0.0.1:46132 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 00:07:11.102378 7340 connection.cc:535] server connection from 127.0.0.1:58416 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 00:07:11.113186 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:11.104288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45828 (local address 127.0.98.60:35537)
0620 00:07:11.104400 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:11.104401 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:11.104408 (+ 7us) server_negotiation.cc:407] Connection header received
0620 00:07:11.104427 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:11.104428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:11.104435 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:11.104480 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:11.105304 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:11.106093 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:11.107376 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:11.107688 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:11.107762 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:11.109523 (+ 1761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:11.109525 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:11.109525 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:11.109532 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:11.111835 (+ 2303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:11.112270 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:11.112271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:11.112272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:11.112337 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:11.112560 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:11.112561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:11.112561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:11.112723 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:11.112832 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:11.112987 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 00:07:11.113084 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 00:07:11.113266 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:11.104082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34202 (local address 127.0.98.61:33203)
0620 00:07:11.104157 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:11.104159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:11.104167 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:07:11.104301 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:11.104302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:11.104318 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:11.104354 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:11.105245 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:11.106091 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:11.107427 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:11.107753 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:11.107792 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:11.109470 (+ 1678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:11.109472 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:11.109472 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:11.109482 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:11.111665 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:11.112320 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:11.112322 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:11.112322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:11.112386 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:11.112609 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:11.112610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:11.112610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:11.112736 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:11.112831 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:11.113059 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 00:07:11.113145 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 00:07:11.121537 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:11.094673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45816 (local address 127.0.98.60:35537)
0620 00:07:11.094722 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:11.094725 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:11.113526 (+ 18801us) server_negotiation.cc:407] Connection header received
0620 00:07:11.113623 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:11.113624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:11.113635 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:11.113667 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:11.114457 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:11.114947 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:11.116081 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:11.116302 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:11.116382 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:11.118105 (+ 1723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:11.118107 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:11.118107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:11.118115 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:11.119840 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:11.120323 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:11.120324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:11.120325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:11.120388 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:11.120640 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:11.120641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:11.120641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:11.120787 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:11.120878 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:11.121205 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 00:07:11.121336 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 00:07:12.090241 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:12.089878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45835 (local address 127.0.98.3:57543)
0620 00:07:12.090042 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:12.090139 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":94,"threads_started":1}
W20260620 00:07:12.090487 7726 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45835 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:45835: connect: Connection refused (error 111)
I20260620 00:07:12.093286 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:12.092977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45835 (local address 127.0.98.2:52423)
0620 00:07:12.093093 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:12.093171 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":74,"threads_started":1}
W20260620 00:07:12.093520 7590 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45835 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:45835: connect: Connection refused (error 111)
I20260620 00:07:12.094816 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:12.094443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45835 (local address 127.0.98.1:49109)
0620 00:07:12.094701 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:12.094777 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":76,"threads_started":1}
W20260620 00:07:12.095171 7453 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45835 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:45835: connect: Connection refused (error 111)
I20260620 00:07:12.593665 7793 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 60dfbd8a9e3a4e4aaad18e7dac6d0ea1)
I20260620 00:07:12.593802 7793 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "b410653b3fa54e709136276f3e32a9bb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35537 } attrs { promote: false } }
I20260620 00:07:12.593861 7794 raft_consensus.cc:482] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 60dfbd8a9e3a4e4aaad18e7dac6d0ea1)
I20260620 00:07:12.593993 7794 raft_consensus.cc:504] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "b410653b3fa54e709136276f3e32a9bb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35537 } attrs { promote: false } }
I20260620 00:07:12.594120 7793 leader_election.cc:288] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 (127.0.98.62:45835), b410653b3fa54e709136276f3e32a9bb (127.0.98.60:35537)
I20260620 00:07:12.594302 7794 leader_election.cc:288] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 (127.0.98.62:45835), 58f4848318664a7586274a478661eed1 (127.0.98.61:33203)
I20260620 00:07:12.594734 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:12.594335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45835 (local address 127.0.0.1:39716)
0620 00:07:12.594627 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:12.594688 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":89,"threads_started":1}
I20260620 00:07:12.594748 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:12.594405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45835 (local address 127.0.0.1:39722)
0620 00:07:12.594678 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:12.594715 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":45,"threads_started":1}
W20260620 00:07:12.594861 7258 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45835: connect: Connection refused (error 111)
W20260620 00:07:12.594869 7071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45835: connect: Connection refused (error 111)
I20260620 00:07:12.595316 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:12.595203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45835 (local address 127.0.0.1:39736)
0620 00:07:12.595238 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:12.595281 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260620 00:07:12.595463 7071 leader_election.cc:334] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 (127.0.98.62:45835): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45835: connect: Connection refused (error 111)
I20260620 00:07:12.595504 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:12.595405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45835 (local address 127.0.0.1:39746)
0620 00:07:12.595454 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:12.595478 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 00:07:12.595561 7258 leader_election.cc:334] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 (127.0.98.62:45835): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45835: connect: Connection refused (error 111)
I20260620 00:07:12.597654 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:12.594561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33203 (local address 127.0.0.1:34218)
0620 00:07:12.595052 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:12.595061 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:12.595116 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:12.595311 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:12.595312 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:12.595314 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:07:12.595364 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:12.595366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:12.596091 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:12.596092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:12.596829 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:12.596831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:12.597421 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:12.597422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:12.597487 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:12.597491 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:07:12.597543 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 00:07:12.597586 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":46,"threads_started":1}
I20260620 00:07:12.597661 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:12.594648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34218 (local address 127.0.98.61:33203)
0620 00:07:12.595074 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:12.595075 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:12.595170 (+ 95us) server_negotiation.cc:407] Connection header received
0620 00:07:12.595194 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:12.595195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:12.595204 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:12.595266 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:07:12.595440 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:12.596000 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:12.596945 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:12.597314 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:12.597429 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:12.597441 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:12.597557 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 00:07:12.597594 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":37,"threads_started":1}
I20260620 00:07:12.597755 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:12.594698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45842 (local address 127.0.98.60:35537)
0620 00:07:12.594891 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:12.594894 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:12.595008 (+ 114us) server_negotiation.cc:407] Connection header received
0620 00:07:12.595041 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:12.595042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:12.595057 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:12.595160 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:07:12.595320 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:12.595953 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:12.596936 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:12.597325 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:12.597454 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:12.597465 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:12.597618 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 00:07:12.597655 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":75,"thread_start_us":38,"threads_started":1}
I20260620 00:07:12.597873 7097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b410653b3fa54e709136276f3e32a9bb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "58f4848318664a7586274a478661eed1" is_pre_election: true
I20260620 00:07:12.597975 7097 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b410653b3fa54e709136276f3e32a9bb in term 1.
I20260620 00:07:12.598142 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:12.594582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35537 (local address 127.0.0.1:45842)
0620 00:07:12.594815 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:12.594825 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:12.594886 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:12.595170 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:12.595171 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:12.595173 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:07:12.595228 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:12.595230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:12.596050 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:12.596051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:12.596820 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:12.596822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:12.597456 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:12.597457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:12.597524 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:12.597527 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:07:12.597609 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 00:07:12.597649 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":40,"threads_started":1}
I20260620 00:07:12.598155 7258 leader_election.cc:302] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58f4848318664a7586274a478661eed1, b410653b3fa54e709136276f3e32a9bb; no voters: 60dfbd8a9e3a4e4aaad18e7dac6d0ea1
I20260620 00:07:12.598271 7794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 00:07:12.598336 7794 raft_consensus.cc:482] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 60dfbd8a9e3a4e4aaad18e7dac6d0ea1)
I20260620 00:07:12.598361 7794 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [term 1 FOLLOWER]: Advancing to term 2
I20260620 00:07:12.598397 7284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "58f4848318664a7586274a478661eed1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b410653b3fa54e709136276f3e32a9bb" is_pre_election: true
I20260620 00:07:12.599223 7794 raft_consensus.cc:504] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "b410653b3fa54e709136276f3e32a9bb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35537 } attrs { promote: false } }
I20260620 00:07:12.599367 7794 leader_election.cc:288] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [CANDIDATE]: Term 2 election: Requested vote from peers 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 (127.0.98.62:45835), 58f4848318664a7586274a478661eed1 (127.0.98.61:33203)
I20260620 00:07:12.599378 7284 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f4848318664a7586274a478661eed1 in current term 2: Already voted for candidate b410653b3fa54e709136276f3e32a9bb in this term.
I20260620 00:07:12.599575 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:12.599484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45835 (local address 127.0.0.1:39754)
0620 00:07:12.599520 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:12.599552 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260620 00:07:12.599593 7071 leader_election.cc:302] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f4848318664a7586274a478661eed1; no voters: 60dfbd8a9e3a4e4aaad18e7dac6d0ea1, b410653b3fa54e709136276f3e32a9bb
I20260620 00:07:12.599774 7793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 00:07:12.599808 7097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b410653b3fa54e709136276f3e32a9bb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "58f4848318664a7586274a478661eed1"
I20260620 00:07:12.599892 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:12.599810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45835 (local address 127.0.0.1:39770)
0620 00:07:12.599842 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:12.599872 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260620 00:07:12.599961 7258 leader_election.cc:334] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 (127.0.98.62:45835): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45835: connect: Connection refused (error 111)
I20260620 00:07:12.600603 7793 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 00:07:12.601316 7097 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b410653b3fa54e709136276f3e32a9bb in term 2.
I20260620 00:07:12.601514 7258 leader_election.cc:302] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58f4848318664a7586274a478661eed1, b410653b3fa54e709136276f3e32a9bb; no voters: 60dfbd8a9e3a4e4aaad18e7dac6d0ea1
I20260620 00:07:12.601663 7794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [term 2 FOLLOWER]: Leader election won for term 2
I20260620 00:07:12.601850 7794 raft_consensus.cc:686] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [term 2 LEADER]: Becoming Leader. State: Replica: b410653b3fa54e709136276f3e32a9bb, State: Running, Role: LEADER
I20260620 00:07:12.601949 7794 consensus_queue.cc:237] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "b410653b3fa54e709136276f3e32a9bb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35537 } attrs { promote: false } }
I20260620 00:07:12.602381 7803 sys_catalog.cc:455] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader b410653b3fa54e709136276f3e32a9bb. Latest consensus state: current_term: 2 leader_uuid: "b410653b3fa54e709136276f3e32a9bb" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "b410653b3fa54e709136276f3e32a9bb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35537 } attrs { promote: false } } }
I20260620 00:07:12.602483 7803 sys_catalog.cc:458] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [sys.catalog]: This master's current role is: LEADER
I20260620 00:07:12.602685 7805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:07:12.603101 7805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:07:12.603163 7805 catalog_manager.cc:1259] Loaded cluster ID: 7bcc619fd37045da9bc9c98085719301
I20260620 00:07:12.603217 7805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:07:12.603444 7805 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:07:12.603556 7805 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb: Loaded TSK: 0
I20260620 00:07:12.603706 7805 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:07:12.611424 7273 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45828:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:07:12.613025 7273 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:07:12.614213 7097 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [term 2 FOLLOWER]: Refusing update from remote peer b410653b3fa54e709136276f3e32a9bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 00:07:12.614251 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:12.614120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45835 (local address 127.0.0.1:39786)
0620 00:07:12.614180 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:12.614215 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 00:07:12.614492 7794 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 00:07:12.614741 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:12.614550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45835 (local address 127.0.0.1:39800)
0620 00:07:12.614669 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:12.614707 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260620 00:07:12.614825 7258 consensus_peers.cc:487] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb -> Peer 60dfbd8a9e3a4e4aaad18e7dac6d0ea1 (127.0.98.62:45835): Couldn't send request to peer 60dfbd8a9e3a4e4aaad18e7dac6d0ea1. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45835: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 00:07:12.615199 7793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b410653b3fa54e709136276f3e32a9bb. Latest consensus state: current_term: 2 leader_uuid: "b410653b3fa54e709136276f3e32a9bb" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "b410653b3fa54e709136276f3e32a9bb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35537 } attrs { promote: false } } }
I20260620 00:07:12.615301 7793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:07:12.615665 7806 mvcc.cc:205] Tried to move back new op lower bound from 7298719877585465344 to 7298719877538537472. Current Snapshot: MvccSnapshot[applied={T|T < 7298719877585465344}]
I20260620 00:07:12.615768 7803 sys_catalog.cc:455] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "b410653b3fa54e709136276f3e32a9bb" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "b410653b3fa54e709136276f3e32a9bb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35537 } attrs { promote: false } } }
I20260620 00:07:12.615882 7803 sys_catalog.cc:458] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb [sys.catalog]: This master's current role is: LEADER
I20260620 00:07:12.615900 7793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "b410653b3fa54e709136276f3e32a9bb" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "60dfbd8a9e3a4e4aaad18e7dac6d0ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } peers { permanent_uuid: "58f4848318664a7586274a478661eed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "b410653b3fa54e709136276f3e32a9bb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35537 } attrs { promote: false } } }
I20260620 00:07:12.615996 7793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58f4848318664a7586274a478661eed1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:07:12.616075 7807 mvcc.cc:205] Tried to move back new op lower bound from 7298719877585465344 to 7298719877538537472. Current Snapshot: MvccSnapshot[applied={T|T < 7298719877585465344}]
I20260620 00:07:12.621754 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:12.618304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46140 (local address 127.0.98.3:46179)
0620 00:07:12.618482 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:12.618484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:12.618613 (+ 129us) server_negotiation.cc:407] Connection header received
0620 00:07:12.618679 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:12.618680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:12.618690 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:12.618753 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:07:12.618927 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:12.619714 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:12.620879 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:12.621321 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:12.621463 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:12.621479 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:12.621641 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 00:07:12.621686 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":95,"threads_started":1}
I20260620 00:07:12.621752 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:12.618246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46179 (local address 127.0.0.1:46140)
0620 00:07:12.618571 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:12.618581 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:12.618626 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:12.618800 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:12.618801 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:12.618805 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:07:12.618861 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:12.618863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:12.619801 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:12.619802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:12.620753 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:12.620755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:12.621464 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:12.621466 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:12.621529 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:12.621533 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:07:12.621612 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 00:07:12.621663 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":3}
I20260620 00:07:12.621971 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:12.618465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58418 (local address 127.0.98.1:44335)
0620 00:07:12.618574 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:12.618576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:12.618736 (+ 160us) server_negotiation.cc:407] Connection header received
0620 00:07:12.618784 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:12.618785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:12.618796 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:12.618852 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:07:12.618992 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:12.619489 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:12.620935 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:12.621562 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:12.621633 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:12.621647 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:12.621843 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 00:07:12.621884 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":66,"threads_started":1}
I20260620 00:07:12.622134 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:12.618299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50088 (local address 127.0.98.2:34161)
0620 00:07:12.618465 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:12.618467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:12.618477 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:07:12.618506 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:12.618507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:12.618518 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:12.618815 (+ 297us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:07:12.618957 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:12.619695 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:12.620940 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:12.621562 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:12.621973 (+ 411us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:12.621992 (+ 19us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:12.622012 (+ 20us) server_negotiation.cc:299] Negotiation successful
0620 00:07:12.622065 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":64,"threads_started":1}
I20260620 00:07:12.621894 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:12.618200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34161 (local address 127.0.0.1:50088)
0620 00:07:12.618291 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:12.618302 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:12.618352 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:12.618822 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:12.618822 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:12.618831 (+ 9us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:07:12.618881 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:12.618882 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:12.619781 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:12.619781 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:12.620848 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:12.620850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:12.621661 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:12.621662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:12.621725 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:12.621729 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:07:12.621762 (+ 33us) client_negotiation.cc:240] Negotiation successful
0620 00:07:12.621808 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":3}
I20260620 00:07:12.622139 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:12.618289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44335 (local address 127.0.0.1:58418)
0620 00:07:12.618665 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:12.618675 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:12.618726 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:12.618894 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:12.618895 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:12.618897 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:07:12.618933 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:12.618935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:12.619577 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:12.619578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:12.620822 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:12.620825 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:12.621673 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:12.621674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:12.621735 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:12.621739 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:07:12.621785 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 00:07:12.621816 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":23,"thread_start_us":96,"threads_started":1}
I20260620 00:07:12.622455 7660 tablet_service.cc:1467] Processing CreateTablet for tablet f1550019ca0a4d1fa290804a428fb0ec (DEFAULT_TABLE table=test-table [id=99fc399b448d4498800f06b1adca9604]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:07:12.622464 7524 tablet_service.cc:1467] Processing CreateTablet for tablet f1550019ca0a4d1fa290804a428fb0ec (DEFAULT_TABLE table=test-table [id=99fc399b448d4498800f06b1adca9604]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:07:12.622570 7387 tablet_service.cc:1467] Processing CreateTablet for tablet f1550019ca0a4d1fa290804a428fb0ec (DEFAULT_TABLE table=test-table [id=99fc399b448d4498800f06b1adca9604]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:07:12.622605 7660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1550019ca0a4d1fa290804a428fb0ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:12.622605 7524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1550019ca0a4d1fa290804a428fb0ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:12.622681 7387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1550019ca0a4d1fa290804a428fb0ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:12.623899 7816 tablet_bootstrap.cc:492] T f1550019ca0a4d1fa290804a428fb0ec P 87af16a18180461ca7afc0bff09802ab: Bootstrap starting.
I20260620 00:07:12.624238 7817 tablet_bootstrap.cc:492] T f1550019ca0a4d1fa290804a428fb0ec P ee53defd13044c57bbb138930c23dbf6: Bootstrap starting.
I20260620 00:07:12.624395 7816 tablet_bootstrap.cc:654] T f1550019ca0a4d1fa290804a428fb0ec P 87af16a18180461ca7afc0bff09802ab: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:12.624571 7817 tablet_bootstrap.cc:654] T f1550019ca0a4d1fa290804a428fb0ec P ee53defd13044c57bbb138930c23dbf6: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:12.624958 7818 tablet_bootstrap.cc:492] T f1550019ca0a4d1fa290804a428fb0ec P 4835482b92894a80985bac108abdac59: Bootstrap starting.
I20260620 00:07:12.625103 7816 tablet_bootstrap.cc:492] T f1550019ca0a4d1fa290804a428fb0ec P 87af16a18180461ca7afc0bff09802ab: No bootstrap required, opened a new log
I20260620 00:07:12.625159 7816 ts_tablet_manager.cc:1397] T f1550019ca0a4d1fa290804a428fb0ec P 87af16a18180461ca7afc0bff09802ab: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:12.625289 7816 raft_consensus.cc:348] T f1550019ca0a4d1fa290804a428fb0ec P 87af16a18180461ca7afc0bff09802ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4835482b92894a80985bac108abdac59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34161 } } peers { permanent_uuid: "ee53defd13044c57bbb138930c23dbf6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44335 } } peers { permanent_uuid: "87af16a18180461ca7afc0bff09802ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46179 } }
I20260620 00:07:12.625402 7816 raft_consensus.cc:374] T f1550019ca0a4d1fa290804a428fb0ec P 87af16a18180461ca7afc0bff09802ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:12.625357 7818 tablet_bootstrap.cc:654] T f1550019ca0a4d1fa290804a428fb0ec P 4835482b92894a80985bac108abdac59: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:12.625433 7816 raft_consensus.cc:729] T f1550019ca0a4d1fa290804a428fb0ec P 87af16a18180461ca7afc0bff09802ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87af16a18180461ca7afc0bff09802ab, State: Initialized, Role: FOLLOWER
I20260620 00:07:12.625474 7816 consensus_queue.cc:260] T f1550019ca0a4d1fa290804a428fb0ec P 87af16a18180461ca7afc0bff09802ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4835482b92894a80985bac108abdac59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34161 } } peers { permanent_uuid: "ee53defd13044c57bbb138930c23dbf6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44335 } } peers { permanent_uuid: "87af16a18180461ca7afc0bff09802ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46179 } }
I20260620 00:07:12.625586 7817 tablet_bootstrap.cc:492] T f1550019ca0a4d1fa290804a428fb0ec P ee53defd13044c57bbb138930c23dbf6: No bootstrap required, opened a new log
I20260620 00:07:12.625617 7816 ts_tablet_manager.cc:1428] T f1550019ca0a4d1fa290804a428fb0ec P 87af16a18180461ca7afc0bff09802ab: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 00:07:12.625645 7817 ts_tablet_manager.cc:1397] T f1550019ca0a4d1fa290804a428fb0ec P ee53defd13044c57bbb138930c23dbf6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:12.625779 7817 raft_consensus.cc:348] T f1550019ca0a4d1fa290804a428fb0ec P ee53defd13044c57bbb138930c23dbf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4835482b92894a80985bac108abdac59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34161 } } peers { permanent_uuid: "ee53defd13044c57bbb138930c23dbf6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44335 } } peers { permanent_uuid: "87af16a18180461ca7afc0bff09802ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46179 } }
I20260620 00:07:12.625845 7817 raft_consensus.cc:374] T f1550019ca0a4d1fa290804a428fb0ec P ee53defd13044c57bbb138930c23dbf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:12.625860 7817 raft_consensus.cc:729] T f1550019ca0a4d1fa290804a428fb0ec P ee53defd13044c57bbb138930c23dbf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee53defd13044c57bbb138930c23dbf6, State: Initialized, Role: FOLLOWER
I20260620 00:07:12.625921 7817 consensus_queue.cc:260] T f1550019ca0a4d1fa290804a428fb0ec P ee53defd13044c57bbb138930c23dbf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4835482b92894a80985bac108abdac59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34161 } } peers { permanent_uuid: "ee53defd13044c57bbb138930c23dbf6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44335 } } peers { permanent_uuid: "87af16a18180461ca7afc0bff09802ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46179 } }
I20260620 00:07:12.626073 7817 ts_tablet_manager.cc:1428] T f1550019ca0a4d1fa290804a428fb0ec P ee53defd13044c57bbb138930c23dbf6: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 00:07:12.626099 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:12.625890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45835 (local address 127.0.98.3:48959)
0620 00:07:12.626022 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:12.626062 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":94,"threads_started":1}
I20260620 00:07:12.626639 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:12.626212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45835 (local address 127.0.98.1:57487)
0620 00:07:12.626564 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:12.626601 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":71,"threads_started":1}
I20260620 00:07:12.626664 7454 heartbeater.cc:503] Master 127.0.98.60:35537 was elected leader, sending a full tablet report...
I20260620 00:07:12.627355 7727 heartbeater.cc:503] Master 127.0.98.60:35537 was elected leader, sending a full tablet report...
I20260620 00:07:12.627314 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:12.626945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45835 (local address 127.0.98.1:36555)
0620 00:07:12.627006 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:12.627035 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 00:07:12.627496 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:12.627268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45835 (local address 127.0.98.3:53499)
0620 00:07:12.627367 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:12.627464 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 00:07:12.627986 7818 tablet_bootstrap.cc:492] T f1550019ca0a4d1fa290804a428fb0ec P 4835482b92894a80985bac108abdac59: No bootstrap required, opened a new log
I20260620 00:07:12.628062 7818 ts_tablet_manager.cc:1397] T f1550019ca0a4d1fa290804a428fb0ec P 4835482b92894a80985bac108abdac59: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 00:07:12.628216 7818 raft_consensus.cc:348] T f1550019ca0a4d1fa290804a428fb0ec P 4835482b92894a80985bac108abdac59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4835482b92894a80985bac108abdac59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34161 } } peers { permanent_uuid: "ee53defd13044c57bbb138930c23dbf6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44335 } } peers { permanent_uuid: "87af16a18180461ca7afc0bff09802ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46179 } }
I20260620 00:07:12.628295 7818 raft_consensus.cc:374] T f1550019ca0a4d1fa290804a428fb0ec P 4835482b92894a80985bac108abdac59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:12.628319 7818 raft_consensus.cc:729] T f1550019ca0a4d1fa290804a428fb0ec P 4835482b92894a80985bac108abdac59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4835482b92894a80985bac108abdac59, State: Initialized, Role: FOLLOWER
I20260620 00:07:12.628376 7818 consensus_queue.cc:260] T f1550019ca0a4d1fa290804a428fb0ec P 4835482b92894a80985bac108abdac59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4835482b92894a80985bac108abdac59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34161 } } peers { permanent_uuid: "ee53defd13044c57bbb138930c23dbf6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44335 } } peers { permanent_uuid: "87af16a18180461ca7afc0bff09802ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46179 } }
I20260620 00:07:12.628536 7818 ts_tablet_manager.cc:1428] T f1550019ca0a4d1fa290804a428fb0ec P 4835482b92894a80985bac108abdac59: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 00:07:12.628921 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:12.628693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45835 (local address 127.0.98.2:59157)
0620 00:07:12.628863 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:12.628893 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":114,"threads_started":1}
I20260620 00:07:12.629146 7591 heartbeater.cc:503] Master 127.0.98.60:35537 was elected leader, sending a full tablet report...
I20260620 00:07:12.629436 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:12.629246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45835 (local address 127.0.98.2:37257)
0620 00:07:12.629311 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:12.629364 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 00:07:12.641410 7824 raft_consensus.cc:482] T f1550019ca0a4d1fa290804a428fb0ec P ee53defd13044c57bbb138930c23dbf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:07:12.641522 7824 raft_consensus.cc:504] T f1550019ca0a4d1fa290804a428fb0ec P ee53defd13044c57bbb138930c23dbf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4835482b92894a80985bac108abdac59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34161 } } peers { permanent_uuid: "ee53defd13044c57bbb138930c23dbf6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44335 } } peers { permanent_uuid: "87af16a18180461ca7afc0bff09802ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46179 } }
I20260620 00:07:12.641831 7824 leader_election.cc:288] T f1550019ca0a4d1fa290804a428fb0ec P ee53defd13044c57bbb138930c23dbf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4835482b92894a80985bac108abdac59 (127.0.98.2:34161), 87af16a18180461ca7afc0bff09802ab (127.0.98.3:46179)
I20260620 00:07:12.646059 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:12.642115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55039 (local address 127.0.98.3:46179)
0620 00:07:12.642176 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:12.642178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:12.642289 (+ 111us) server_negotiation.cc:407] Connection header received
0620 00:07:12.642324 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:12.642325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:12.642338 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:12.642407 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:07:12.642624 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:12.643376 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:12.645019 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:12.645673 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:12.645739 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:12.645754 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:12.645937 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 00:07:12.645979 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 00:07:12.646059 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:12.642124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45835 (local address 127.0.98.2:34161)
0620 00:07:12.642176 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:12.642178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:12.642188 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:07:12.642304 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:12.642305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:12.642361 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:12.642419 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:07:12.642623 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:12.643406 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:12.645019 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:12.645639 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:12.645752 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:12.645765 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:12.645937 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 00:07:12.645979 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 00:07:12.646087 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:12.642000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34161 (local address 127.0.98.1:45835)
0620 00:07:12.642079 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:12.642093 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:12.642151 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:12.642432 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:12.642433 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:12.642436 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:07:12.642549 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:12.642551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:12.643502 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:12.643503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:12.644893 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:12.644897 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:12.645788 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:12.645789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:12.645855 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:12.645857 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 00:07:12.645920 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 00:07:12.645979 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":71}
I20260620 00:07:12.646098 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:12.642041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46179 (local address 127.0.98.1:55039)
0620 00:07:12.642221 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:12.642231 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:12.642283 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:12.642432 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:12.642433 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:12.642436 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:07:12.642549 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:12.642551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:12.643520 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:12.643521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:12.644893 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:12.644897 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:12.645760 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:12.645761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:12.645844 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:12.645851 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 00:07:12.645920 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 00:07:12.645979 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":88,"thread_start_us":119,"threads_started":1}
I20260620 00:07:12.646432 7544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1550019ca0a4d1fa290804a428fb0ec" candidate_uuid: "ee53defd13044c57bbb138930c23dbf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4835482b92894a80985bac108abdac59" is_pre_election: true
I20260620 00:07:12.646543 7544 raft_consensus.cc:2454] T f1550019ca0a4d1fa290804a428fb0ec P 4835482b92894a80985bac108abdac59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee53defd13044c57bbb138930c23dbf6 in term 0.
I20260620 00:07:12.646593 7680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1550019ca0a4d1fa290804a428fb0ec" candidate_uuid: "ee53defd13044c57bbb138930c23dbf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87af16a18180461ca7afc0bff09802ab" is_pre_election: true
I20260620 00:07:12.646728 7680 raft_consensus.cc:2454] T f1550019ca0a4d1fa290804a428fb0ec P 87af16a18180461ca7afc0bff09802ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee53defd13044c57bbb138930c23dbf6 in term 0.
I20260620 00:07:12.646818 7343 leader_election.cc:302] T f1550019ca0a4d1fa290804a428fb0ec P ee53defd13044c57bbb138930c23dbf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4835482b92894a80985bac108abdac59, ee53defd13044c57bbb138930c23dbf6; no voters:
I20260620 00:07:12.646970 7824 raft_consensus.cc:2781] T f1550019ca0a4d1fa290804a428fb0ec P ee53defd13044c57bbb138930c23dbf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:07:12.647012 7824 raft_consensus.cc:482] T f1550019ca0a4d1fa290804a428fb0ec P ee53defd13044c57bbb138930c23dbf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:07:12.647034 7824 raft_consensus.cc:3037] T f1550019ca0a4d1fa290804a428fb0ec P ee53defd13044c57bbb138930c23dbf6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:12.647856 7824 raft_consensus.cc:504] T f1550019ca0a4d1fa290804a428fb0ec P ee53defd13044c57bbb138930c23dbf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4835482b92894a80985bac108abdac59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34161 } } peers { permanent_uuid: "ee53defd13044c57bbb138930c23dbf6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44335 } } peers { permanent_uuid: "87af16a18180461ca7afc0bff09802ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46179 } }
I20260620 00:07:12.648046 7824 leader_election.cc:288] T f1550019ca0a4d1fa290804a428fb0ec P ee53defd13044c57bbb138930c23dbf6 [CANDIDATE]: Term 1 election: Requested vote from peers 4835482b92894a80985bac108abdac59 (127.0.98.2:34161), 87af16a18180461ca7afc0bff09802ab (127.0.98.3:46179)
I20260620 00:07:12.648209 7680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1550019ca0a4d1fa290804a428fb0ec" candidate_uuid: "ee53defd13044c57bbb138930c23dbf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87af16a18180461ca7afc0bff09802ab"
I20260620 00:07:12.648294 7680 raft_consensus.cc:3037] T f1550019ca0a4d1fa290804a428fb0ec P 87af16a18180461ca7afc0bff09802ab [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:12.648270 7544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1550019ca0a4d1fa290804a428fb0ec" candidate_uuid: "ee53defd13044c57bbb138930c23dbf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4835482b92894a80985bac108abdac59"
I20260620 00:07:12.648378 7544 raft_consensus.cc:3037] T f1550019ca0a4d1fa290804a428fb0ec P 4835482b92894a80985bac108abdac59 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:12.648998 7544 raft_consensus.cc:2454] T f1550019ca0a4d1fa290804a428fb0ec P 4835482b92894a80985bac108abdac59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee53defd13044c57bbb138930c23dbf6 in term 1.
I20260620 00:07:12.649012 7680 raft_consensus.cc:2454] T f1550019ca0a4d1fa290804a428fb0ec P 87af16a18180461ca7afc0bff09802ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee53defd13044c57bbb138930c23dbf6 in term 1.
I20260620 00:07:12.649212 7342 leader_election.cc:302] T f1550019ca0a4d1fa290804a428fb0ec P ee53defd13044c57bbb138930c23dbf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87af16a18180461ca7afc0bff09802ab, ee53defd13044c57bbb138930c23dbf6; no voters:
I20260620 00:07:12.649333 7824 raft_consensus.cc:2781] T f1550019ca0a4d1fa290804a428fb0ec P ee53defd13044c57bbb138930c23dbf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:07:12.649516 7824 raft_consensus.cc:686] T f1550019ca0a4d1fa290804a428fb0ec P ee53defd13044c57bbb138930c23dbf6 [term 1 LEADER]: Becoming Leader. State: Replica: ee53defd13044c57bbb138930c23dbf6, State: Running, Role: LEADER
I20260620 00:07:12.649605 7824 consensus_queue.cc:237] T f1550019ca0a4d1fa290804a428fb0ec P ee53defd13044c57bbb138930c23dbf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4835482b92894a80985bac108abdac59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34161 } } peers { permanent_uuid: "ee53defd13044c57bbb138930c23dbf6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44335 } } peers { permanent_uuid: "87af16a18180461ca7afc0bff09802ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46179 } }
I20260620 00:07:12.649996 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:12.649881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45835 (local address 127.0.98.1:33279)
0620 00:07:12.649922 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:12.649965 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260620 00:07:12.650481 7274 catalog_manager.cc:5495] T f1550019ca0a4d1fa290804a428fb0ec P ee53defd13044c57bbb138930c23dbf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee53defd13044c57bbb138930c23dbf6 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ee53defd13044c57bbb138930c23dbf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4835482b92894a80985bac108abdac59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee53defd13044c57bbb138930c23dbf6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87af16a18180461ca7afc0bff09802ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46179 } health_report { overall_health: UNKNOWN } } }
I20260620 00:07:12.659950 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:12.657604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58424 (local address 127.0.98.1:44335)
0620 00:07:12.657662 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:12.657664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:12.657672 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:07:12.657706 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:12.657707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:12.657726 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:12.657800 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:07:12.658001 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:12.658501 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:12.659232 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:12.659426 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:12.659482 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:12.659606 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:07:12.659708 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:07:12.659764 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:12.659799 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 00:07:12.659816 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 00:07:12.661245 7544 raft_consensus.cc:1261] T f1550019ca0a4d1fa290804a428fb0ec P 4835482b92894a80985bac108abdac59 [term 1 FOLLOWER]: Refusing update from remote peer ee53defd13044c57bbb138930c23dbf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:07:12.661320 7680 raft_consensus.cc:1261] T f1550019ca0a4d1fa290804a428fb0ec P 87af16a18180461ca7afc0bff09802ab [term 1 FOLLOWER]: Refusing update from remote peer ee53defd13044c57bbb138930c23dbf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:07:12.661641 7824 consensus_queue.cc:1035] T f1550019ca0a4d1fa290804a428fb0ec P ee53defd13044c57bbb138930c23dbf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4835482b92894a80985bac108abdac59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:07:12.661800 7829 consensus_queue.cc:1035] T f1550019ca0a4d1fa290804a428fb0ec P ee53defd13044c57bbb138930c23dbf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87af16a18180461ca7afc0bff09802ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:07:12.662751 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:12.662626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45835 (local address 127.0.98.3:37019)
0620 00:07:12.662668 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:12.662712 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260620 00:07:12.663579 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:12.662754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45835 (local address 127.0.98.2:47287)
0620 00:07:12.663504 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:12.663538 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":734}
I20260620 00:07:12.664060 7833 mvcc.cc:205] Tried to move back new op lower bound from 7298719877777977344 to 7298719877733576704. Current Snapshot: MvccSnapshot[applied={T|T < 7298719877777977344}]
I20260620 00:07:12.664440 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:12.663471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45835 (local address 127.0.98.1:50997)
0620 00:07:12.664374 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:12.664408 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":886}
I20260620 00:07:12.664786 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:12.664704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45835 (local address 127.0.98.1:35945)
0620 00:07:12.664737 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:12.664759 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260620 00:07:12.664997 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:12.664898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45835 (local address 127.0.98.3:35589)
0620 00:07:12.664934 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:12.664964 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260620 00:07:12.665531 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:12.664290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45835 (local address 127.0.98.2:36057)
0620 00:07:12.665473 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:12.665508 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1166}
I20260620 00:07:12.667526 7273 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:07:12.667640 7273 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:07:12.668932 7273 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b410653b3fa54e709136276f3e32a9bb: Sending DeleteTablet for 3 replicas of tablet f1550019ca0a4d1fa290804a428fb0ec
I20260620 00:07:12.669240 7660 tablet_service.cc:1514] Processing DeleteTablet for tablet f1550019ca0a4d1fa290804a428fb0ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:07:12 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:46140
I20260620 00:07:12.669245 7524 tablet_service.cc:1514] Processing DeleteTablet for tablet f1550019ca0a4d1fa290804a428fb0ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:07:12 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:50088
I20260620 00:07:12.669301 7387 tablet_service.cc:1514] Processing DeleteTablet for tablet f1550019ca0a4d1fa290804a428fb0ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:07:12 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:58418
I20260620 00:07:12.669656 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 7235
I20260620 00:07:12.672102 7842 tablet_replica.cc:330] T f1550019ca0a4d1fa290804a428fb0ec P 87af16a18180461ca7afc0bff09802ab: stopping tablet replica
I20260620 00:07:12.672168 7842 raft_consensus.cc:2229] T f1550019ca0a4d1fa290804a428fb0ec P 87af16a18180461ca7afc0bff09802ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:07:12.672253 7842 raft_consensus.cc:2258] T f1550019ca0a4d1fa290804a428fb0ec P 87af16a18180461ca7afc0bff09802ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:07:12.673653 7842 ts_tablet_manager.cc:1905] T f1550019ca0a4d1fa290804a428fb0ec P 87af16a18180461ca7afc0bff09802ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:07:12.675741 7842 ts_tablet_manager.cc:1918] T f1550019ca0a4d1fa290804a428fb0ec P 87af16a18180461ca7afc0bff09802ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:07:12.675814 7842 log.cc:1196] T f1550019ca0a4d1fa290804a428fb0ec P 87af16a18180461ca7afc0bff09802ab: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914001714546-392-0/minicluster-data/ts-2/wal/wals/f1550019ca0a4d1fa290804a428fb0ec
I20260620 00:07:12.676057 7842 ts_tablet_manager.cc:1939] T f1550019ca0a4d1fa290804a428fb0ec P 87af16a18180461ca7afc0bff09802ab: Deleting consensus metadata
I20260620 00:07:12.676294 7843 tablet_replica.cc:330] T f1550019ca0a4d1fa290804a428fb0ec P 4835482b92894a80985bac108abdac59: stopping tablet replica
I20260620 00:07:12.676362 7843 raft_consensus.cc:2229] T f1550019ca0a4d1fa290804a428fb0ec P 4835482b92894a80985bac108abdac59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:07:12.676411 7843 raft_consensus.cc:2258] T f1550019ca0a4d1fa290804a428fb0ec P 4835482b92894a80985bac108abdac59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:07:12.676550 7260 catalog_manager.cc:4841] TS 87af16a18180461ca7afc0bff09802ab (127.0.98.3:46179): tablet f1550019ca0a4d1fa290804a428fb0ec (table test-table [id=99fc399b448d4498800f06b1adca9604]) successfully deleted
W20260620 00:07:12.676867 7813 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:12.676291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44335 (local address 127.0.0.1:58428)
0620 00:07:12.676352 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:12.676364 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:12.676428 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:12.676815 (+ 387us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44335: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 00:07:12.676891 7843 ts_tablet_manager.cc:1905] T f1550019ca0a4d1fa290804a428fb0ec P 4835482b92894a80985bac108abdac59: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 00:07:12.677022 7261 catalog_manager.cc:4584] TS ee53defd13044c57bbb138930c23dbf6 (127.0.98.1:44335): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f1550019ca0a4d1fa290804a428fb0ec: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44335: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 00:07:12.677052 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 7461
I20260620 00:07:12.682132 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:12.681984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34161 (local address 127.0.0.1:50092)
0620 00:07:12.682045 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:12.682078 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 00:07:12.682159 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 7597
I20260620 00:07:12.688248 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 6953
I20260620 00:07:12.693193 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 7141
2026-06-20T00:07:12Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4740 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:07:12 dist-test-slave-grqt krb5kdc[7847](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:07:12 dist-test-slave-grqt krb5kdc[7847](info): set up 2 sockets
Jun 20 00:07:12 dist-test-slave-grqt krb5kdc[7847](info): commencing operation
krb5kdc: starting...
W20260620 00:07:13.746063 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:07:13 dist-test-slave-grqt krb5kdc[7847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:07:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:07:13Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260620 00:07:13.878170 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38645
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:07:13.962666 7860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:13.962889 7860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:13.962915 7860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:13.964186 7860 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:07:13.964242 7860 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 00:07:13.964257 7860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:13.964270 7860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:13.964283 7860 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:07:13.964298 7860 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:07:13.965940 7860 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38645
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:13.966218 7860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:13.966459 7860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:13.971575 7868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:13.971691 7865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:13.971601 7860 server_base.cc:1034] running on GCE node
W20260620 00:07:13.971594 7866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:13.972294 7860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:13.972617 7860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:13.973754 7860 hybrid_clock.cc:648] HybridClock initialized: now 1781914033973731 us; error 40 us; skew 500 ppm
Jun 20 00:07:13 dist-test-slave-grqt krb5kdc[7847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914033, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:13.976369 7860 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260620 00:07:13.976843 7860 webserver.cc:460] Webserver started at http://127.0.98.62:39515/ using document root <none> and password file <none>
I20260620 00:07:13.977030 7860 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:13.977072 7860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:13.977164 7860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:13.978269 7860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "05cccc5116e84b959eee49c8c6ca05a4"
format_stamp: "Formatted at 2026-06-20 00:07:13 on dist-test-slave-grqt"
server_key: "2a3d8ab6a8a303829c1d3182f82323dd"
server_key_iv: "5f695338e7e6b8eb38ed4c9fe998315c"
server_key_version: "encryptionkey@0"
I20260620 00:07:13.978580 7860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "05cccc5116e84b959eee49c8c6ca05a4"
format_stamp: "Formatted at 2026-06-20 00:07:13 on dist-test-slave-grqt"
server_key: "2a3d8ab6a8a303829c1d3182f82323dd"
server_key_iv: "5f695338e7e6b8eb38ed4c9fe998315c"
server_key_version: "encryptionkey@0"
I20260620 00:07:13.979903 7860 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 00:07:13.980654 7875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:13.980865 7860 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:07:13.980954 7860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "05cccc5116e84b959eee49c8c6ca05a4"
format_stamp: "Formatted at 2026-06-20 00:07:13 on dist-test-slave-grqt"
server_key: "2a3d8ab6a8a303829c1d3182f82323dd"
server_key_iv: "5f695338e7e6b8eb38ed4c9fe998315c"
server_key_version: "encryptionkey@0"
I20260620 00:07:13.981014 7860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:13.996958 7860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:13.999423 7860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:13.999545 7860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:14.004277 7860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38645
I20260620 00:07:14.004473 7937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38645 every 8 connection(s)
I20260620 00:07:14.004751 7860 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:07:14.005504 7938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:14.007779 7938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05cccc5116e84b959eee49c8c6ca05a4: Bootstrap starting.
I20260620 00:07:14.008406 7938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05cccc5116e84b959eee49c8c6ca05a4: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:14.008733 7938 log.cc:824] T 00000000000000000000000000000000 P 05cccc5116e84b959eee49c8c6ca05a4: Log is configured to *not* fsync() on all Append() calls
I20260620 00:07:14.009487 7938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05cccc5116e84b959eee49c8c6ca05a4: No bootstrap required, opened a new log
I20260620 00:07:14.011565 7938 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05cccc5116e84b959eee49c8c6ca05a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05cccc5116e84b959eee49c8c6ca05a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38645 } }
I20260620 00:07:14.011708 7938 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05cccc5116e84b959eee49c8c6ca05a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:14.011731 7938 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05cccc5116e84b959eee49c8c6ca05a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05cccc5116e84b959eee49c8c6ca05a4, State: Initialized, Role: FOLLOWER
I20260620 00:07:14.011905 7938 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05cccc5116e84b959eee49c8c6ca05a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05cccc5116e84b959eee49c8c6ca05a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38645 } }
I20260620 00:07:14.011981 7938 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05cccc5116e84b959eee49c8c6ca05a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:07:14.012003 7938 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05cccc5116e84b959eee49c8c6ca05a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:07:14.012058 7938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05cccc5116e84b959eee49c8c6ca05a4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:14.012599 7938 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05cccc5116e84b959eee49c8c6ca05a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05cccc5116e84b959eee49c8c6ca05a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38645 } }
I20260620 00:07:14.012715 7938 leader_election.cc:302] T 00000000000000000000000000000000 P 05cccc5116e84b959eee49c8c6ca05a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05cccc5116e84b959eee49c8c6ca05a4; no voters:
I20260620 00:07:14.012909 7938 leader_election.cc:288] T 00000000000000000000000000000000 P 05cccc5116e84b959eee49c8c6ca05a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:07:14.012956 7941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05cccc5116e84b959eee49c8c6ca05a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:07:14.013147 7941 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05cccc5116e84b959eee49c8c6ca05a4 [term 1 LEADER]: Becoming Leader. State: Replica: 05cccc5116e84b959eee49c8c6ca05a4, State: Running, Role: LEADER
I20260620 00:07:14.013190 7938 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05cccc5116e84b959eee49c8c6ca05a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:07:14.013335 7941 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05cccc5116e84b959eee49c8c6ca05a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05cccc5116e84b959eee49c8c6ca05a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38645 } }
I20260620 00:07:14.013660 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 7860
I20260620 00:07:14.013748 7942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05cccc5116e84b959eee49c8c6ca05a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05cccc5116e84b959eee49c8c6ca05a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05cccc5116e84b959eee49c8c6ca05a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38645 } } }
I20260620 00:07:14.013785 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:07:14.013824 7942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05cccc5116e84b959eee49c8c6ca05a4 [sys.catalog]: This master's current role is: LEADER
I20260620 00:07:14.014046 392 external_mini_cluster.cc:1428] Setting key 0017a09c828929a8b6371ba8d20909f7
I20260620 00:07:14.014050 7943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05cccc5116e84b959eee49c8c6ca05a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05cccc5116e84b959eee49c8c6ca05a4. Latest consensus state: current_term: 1 leader_uuid: "05cccc5116e84b959eee49c8c6ca05a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05cccc5116e84b959eee49c8c6ca05a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38645 } } }
I20260620 00:07:14.014111 7943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05cccc5116e84b959eee49c8c6ca05a4 [sys.catalog]: This master's current role is: LEADER
I20260620 00:07:14.014220 7949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:07:14.014966 7949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:07:14.016698 7949 catalog_manager.cc:1347] Generated new cluster ID: d1b16a5af99f4531a62f3090f519f80d
I20260620 00:07:14.016757 7949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 00:07:14 dist-test-slave-grqt krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 00:07:14.026292 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:14.014781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35538 (local address 127.0.98.62:38645)
0620 00:07:14.014954 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:14.014965 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:14.014999 (+ 34us) server_negotiation.cc:407] Connection header received
0620 00:07:14.015168 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:14.015177 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:14.015236 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:14.015406 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:14.016179 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.017701 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.018797 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.019422 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.019527 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:14.022431 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:14.022436 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:14.022436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:14.022450 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:14.024596 (+ 2146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:14.024986 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:14.024990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:14.024990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:14.025108 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:14.025322 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:14.025323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:14.025323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:14.025676 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:14.025838 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:14.025920 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 00:07:14.026064 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":77,"threads_started":1}
I20260620 00:07:14.028440 7949 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:07:14.029147 7949 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:07:14.037921 7949 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05cccc5116e84b959eee49c8c6ca05a4: Generated new TSK 0
I20260620 00:07:14.038146 7949 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260620 00:07:14.073486 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38645
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:07:14.159196 7964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:14.159443 7964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:14.159483 7964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:14.160681 7964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:14.160755 7964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:14.160791 7964 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 00:07:14.160832 7964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:07:14.162346 7964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:14.162632 7964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:14.162885 7964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:14.169409 7970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:14.169450 7969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:14.169569 7964 server_base.cc:1034] running on GCE node
W20260620 00:07:14.169646 7972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:14.169878 7964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:14.170150 7964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:14.171334 7964 hybrid_clock.cc:648] HybridClock initialized: now 1781914034171311 us; error 38 us; skew 500 ppm
Jun 20 00:07:14 dist-test-slave-grqt krb5kdc[7847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914034, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:14.173840 7964 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 00:07:14.174297 7964 webserver.cc:460] Webserver started at http://127.0.98.1:44637/ using document root <none> and password file <none>
I20260620 00:07:14.174530 7964 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:14.174599 7964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:14.174736 7964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:14.175906 7964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-0/data/instance:
uuid: "6f34094d8e684686b4004cf910f8f16a"
format_stamp: "Formatted at 2026-06-20 00:07:14 on dist-test-slave-grqt"
server_key: "aaee6218e273c2e403e1182e2b3d127c"
server_key_iv: "0d613c8235ee8ae61acb8900969093e4"
server_key_version: "encryptionkey@0"
I20260620 00:07:14.176273 7964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6f34094d8e684686b4004cf910f8f16a"
format_stamp: "Formatted at 2026-06-20 00:07:14 on dist-test-slave-grqt"
server_key: "aaee6218e273c2e403e1182e2b3d127c"
server_key_iv: "0d613c8235ee8ae61acb8900969093e4"
server_key_version: "encryptionkey@0"
I20260620 00:07:14.177752 7964 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:07:14.178635 7979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:14.178896 7964 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:14.179004 7964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-0/wal
uuid: "6f34094d8e684686b4004cf910f8f16a"
format_stamp: "Formatted at 2026-06-20 00:07:14 on dist-test-slave-grqt"
server_key: "aaee6218e273c2e403e1182e2b3d127c"
server_key_iv: "0d613c8235ee8ae61acb8900969093e4"
server_key_version: "encryptionkey@0"
I20260620 00:07:14.179092 7964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:14.186031 7964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:14.188127 7964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:14.188301 7964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:14.188819 7964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:07:14.188897 7964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:14.188946 7964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:07:14.188971 7964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 00:07:14 dist-test-slave-grqt krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914034, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 00:07:14.196560 7964 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35827
I20260620 00:07:14.196635 8094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35827 every 8 connection(s)
I20260620 00:07:14.196975 7964 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:07:14.199206 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 7964
I20260620 00:07:14.199342 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:07:14.199600 392 external_mini_cluster.cc:1428] Setting key 80c44832c859e8ce29cb320401173856
I20260620 00:07:14.199949 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:14.189399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47263 (local address 127.0.98.62:38645)
0620 00:07:14.189457 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:14.189460 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:14.189731 (+ 271us) server_negotiation.cc:407] Connection header received
0620 00:07:14.190496 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:14.190497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:14.190507 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:14.190619 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:14.191615 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.192149 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.193850 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.194145 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.194245 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:14.196851 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:14.196853 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:14.196853 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:14.196860 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:14.198302 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:14.198759 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:14.198760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:14.198760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:14.198819 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:14.199076 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:14.199077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:14.199078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:14.199208 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:14.199343 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:14.199723 (+ 380us) server_negotiation.cc:299] Negotiation successful
0620 00:07:14.199848 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 00:07:14.200111 7988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:14.189361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38645 (local address 127.0.98.1:47263)
0620 00:07:14.189557 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:14.189605 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:14.190348 (+ 743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:14.190713 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:14.190718 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:14.191144 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:14.191499 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:14.191507 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.192225 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.192226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:14.193747 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:14.193751 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.194275 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.194276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:14.194398 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:14.194972 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:14.194981 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:14.196750 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:14.198410 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:14.198416 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:14.198418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:14.198642 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:14.198892 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:14.198892 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:14.198892 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:14.198990 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:14.199339 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:14.199346 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:14.199522 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 00:07:14.199648 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 00:07:14.199834 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
I20260620 00:07:14.200538 8095 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38645
I20260620 00:07:14.200624 8095 heartbeater.cc:464] Registering TS with master...
I20260620 00:07:14.200831 8095 heartbeater.cc:511] Master 127.0.98.62:38645 requested a full tablet report, sending...
I20260620 00:07:14.201427 7891 ts_manager.cc:194] Registered new tserver with Master: 6f34094d8e684686b4004cf910f8f16a (127.0.98.1:35827)
I20260620 00:07:14.202179 7891 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47263
I20260620 00:07:14.208868 8102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:14.201692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38645 (local address 127.0.98.1:49845)
0620 00:07:14.201839 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:14.201853 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:14.201921 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:14.202504 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:14.202505 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:14.202759 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:14.202832 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:14.202835 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.203795 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.203796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:14.204411 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:14.204414 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.204928 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.204929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:14.204977 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:14.205552 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:14.205562 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:14.205895 (+ 333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:14.207781 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:14.207782 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:14.207783 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:14.207989 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:14.208222 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:14.208222 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:14.208223 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:14.208277 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:14.208580 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:14.208581 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:14.208632 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 00:07:14.208686 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 00:07:14.208776 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":80,"threads_started":1}
I20260620 00:07:14.208927 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:14.201705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49845 (local address 127.0.98.62:38645)
0620 00:07:14.202388 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:14.202390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:14.202399 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:07:14.202421 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:14.202421 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:14.202431 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:14.202495 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:14.202950 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.203679 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.204516 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.204830 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.204910 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:14.205988 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:14.205990 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:14.205991 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:14.205999 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:14.207679 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:14.208076 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:14.208078 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:14.208078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:14.208141 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:14.208345 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:14.208346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:14.208347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:14.208505 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:14.208591 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:14.208737 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 00:07:14.208832 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260620 00:07:14.234081 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38645
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:07:14.317992 8104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:14.318228 8104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:14.318254 8104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:14.319424 8104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:14.319473 8104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:14.319494 8104 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 00:07:14.319512 8104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:07:14.321063 8104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:14.321285 8104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:14.321501 8104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 00:07:14.326818 8104 server_base.cc:1034] running on GCE node
W20260620 00:07:14.326784 8110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:14.326766 8112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:14.326787 8109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:14.327133 8104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:14.327350 8104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:14.328553 8104 hybrid_clock.cc:648] HybridClock initialized: now 1781914034328542 us; error 32 us; skew 500 ppm
Jun 20 00:07:14 dist-test-slave-grqt krb5kdc[7847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914034, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:14.331255 8104 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 00:07:14.331782 8104 webserver.cc:460] Webserver started at http://127.0.98.2:37307/ using document root <none> and password file <none>
I20260620 00:07:14.332028 8104 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:14.332074 8104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:14.332244 8104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:14.333359 8104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-1/data/instance:
uuid: "69ca88ea258443a39dad68029d288f95"
format_stamp: "Formatted at 2026-06-20 00:07:14 on dist-test-slave-grqt"
server_key: "5f39cdfa70ec58d1beb58ce9c4e86a3b"
server_key_iv: "4541f3072a2560bed2fdb203a27d29b4"
server_key_version: "encryptionkey@0"
I20260620 00:07:14.333859 8104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-1/wal/instance:
uuid: "69ca88ea258443a39dad68029d288f95"
format_stamp: "Formatted at 2026-06-20 00:07:14 on dist-test-slave-grqt"
server_key: "5f39cdfa70ec58d1beb58ce9c4e86a3b"
server_key_iv: "4541f3072a2560bed2fdb203a27d29b4"
server_key_version: "encryptionkey@0"
I20260620 00:07:14.335294 8104 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:07:14.336141 8119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:14.336376 8104 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:14.336478 8104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-1/wal
uuid: "69ca88ea258443a39dad68029d288f95"
format_stamp: "Formatted at 2026-06-20 00:07:14 on dist-test-slave-grqt"
server_key: "5f39cdfa70ec58d1beb58ce9c4e86a3b"
server_key_iv: "4541f3072a2560bed2fdb203a27d29b4"
server_key_version: "encryptionkey@0"
I20260620 00:07:14.336551 8104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:14.354257 8104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:14.356664 8104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:14.356858 8104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:14.357496 8104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:07:14.357559 8104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:14.357601 8104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:07:14.357623 8104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 00:07:14 dist-test-slave-grqt krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914034, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 00:07:14.365314 8104 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34395
I20260620 00:07:14.365550 8234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34395 every 8 connection(s)
I20260620 00:07:14.365898 8104 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:07:14.368685 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:14.358157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46521 (local address 127.0.98.62:38645)
0620 00:07:14.358231 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:14.358233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:14.358426 (+ 193us) server_negotiation.cc:407] Connection header received
0620 00:07:14.358959 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:14.358960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:14.358973 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:14.359030 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:14.359792 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.360310 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.361362 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.361710 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.361802 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:14.364822 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:14.364823 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:14.364824 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:14.364833 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:14.366666 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:14.367117 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:14.367119 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:14.367119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:14.367216 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:14.367531 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:14.367532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:14.367532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:14.367667 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:14.368071 (+ 404us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:14.368480 (+ 409us) server_negotiation.cc:299] Negotiation successful
0620 00:07:14.368579 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 00:07:14.368945 8128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:14.358113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38645 (local address 127.0.98.2:46521)
0620 00:07:14.358289 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:14.358330 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:14.358844 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:14.359111 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:14.359115 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:14.359479 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:14.359692 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:14.359697 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.360413 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.360414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:14.361248 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:14.361250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.361813 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.361814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:14.361971 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:14.362600 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:14.362613 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:14.364659 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:14.366775 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:14.366781 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:14.366783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:14.367014 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:14.367306 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:14.367307 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:14.367307 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:14.367420 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:14.368062 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:14.368069 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:14.368234 (+ 165us) client_negotiation.cc:769] Sending connection context
0620 00:07:14.368448 (+ 214us) client_negotiation.cc:240] Negotiation successful
0620 00:07:14.368646 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":106,"threads_started":1}
I20260620 00:07:14.369493 8235 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38645
I20260620 00:07:14.369602 8235 heartbeater.cc:464] Registering TS with master...
I20260620 00:07:14.369607 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 8104
I20260620 00:07:14.369741 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:07:14.369848 8235 heartbeater.cc:511] Master 127.0.98.62:38645 requested a full tablet report, sending...
I20260620 00:07:14.370008 392 external_mini_cluster.cc:1428] Setting key 7513e7d05ac672fb949fa6c3eec24011
I20260620 00:07:14.370291 7892 ts_manager.cc:194] Registered new tserver with Master: 69ca88ea258443a39dad68029d288f95 (127.0.98.2:34395)
I20260620 00:07:14.370852 7892 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46521
I20260620 00:07:14.377960 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:14.370616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33763 (local address 127.0.98.62:38645)
0620 00:07:14.370657 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:14.370659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:14.370769 (+ 110us) server_negotiation.cc:407] Connection header received
0620 00:07:14.370851 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:14.370852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:14.370863 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:14.370938 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:14.371298 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.372134 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.373493 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.373692 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.373817 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:14.374918 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:14.374920 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:14.374920 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:14.374927 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:14.376646 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:14.377023 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:14.377024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:14.377025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:14.377082 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:14.377297 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:14.377298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:14.377299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:14.377519 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:14.377608 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:14.377789 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 00:07:14.377861 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 00:07:14.377974 8241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:14.370562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38645 (local address 127.0.98.2:33763)
0620 00:07:14.370715 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:14.370725 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:14.370794 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:14.370944 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:14.370944 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:14.371160 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:14.371212 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:14.371214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.372272 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.372273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:14.373208 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:14.373212 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.373820 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.373821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:14.373894 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:14.374425 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:14.374429 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:14.374811 (+ 382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:14.376754 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:14.376755 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:14.376755 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:14.376931 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:14.377150 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:14.377151 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:14.377151 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:14.377214 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:14.377599 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:14.377600 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:14.377650 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 00:07:14.377742 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 00:07:14.377861 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":74,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 00:07:14.401114 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38645
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:07:14.486989 8244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:14.487228 8244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:14.487257 8244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:14.488480 8244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:14.488529 8244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:14.488551 8244 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 00:07:14.488570 8244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:07:14.490212 8244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:14.490458 8244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:14.490656 8244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:14.496733 8249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:14.496760 8252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:14.496892 8244 server_base.cc:1034] running on GCE node
W20260620 00:07:14.497049 8250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:14.497337 8244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:14.497977 8244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:14.499177 8244 hybrid_clock.cc:648] HybridClock initialized: now 1781914034499120 us; error 73 us; skew 500 ppm
Jun 20 00:07:14 dist-test-slave-grqt krb5kdc[7847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914034, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:14.507711 8244 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 00:07:14.508258 8244 webserver.cc:460] Webserver started at http://127.0.98.3:45933/ using document root <none> and password file <none>
I20260620 00:07:14.508517 8244 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:14.508570 8244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:14.508711 8244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:14.510035 8244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-2/data/instance:
uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4"
format_stamp: "Formatted at 2026-06-20 00:07:14 on dist-test-slave-grqt"
server_key: "81b9bf811a6c7a57dc9a1c48b929c306"
server_key_iv: "f64ec6f9b0e6d57a301f8c9c577286d5"
server_key_version: "encryptionkey@0"
I20260620 00:07:14.510452 8244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4"
format_stamp: "Formatted at 2026-06-20 00:07:14 on dist-test-slave-grqt"
server_key: "81b9bf811a6c7a57dc9a1c48b929c306"
server_key_iv: "f64ec6f9b0e6d57a301f8c9c577286d5"
server_key_version: "encryptionkey@0"
I20260620 00:07:14.511977 8244 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:07:14.512876 8259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:14.513131 8244 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:14.513244 8244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-2/wal
uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4"
format_stamp: "Formatted at 2026-06-20 00:07:14 on dist-test-slave-grqt"
server_key: "81b9bf811a6c7a57dc9a1c48b929c306"
server_key_iv: "f64ec6f9b0e6d57a301f8c9c577286d5"
server_key_version: "encryptionkey@0"
I20260620 00:07:14.513420 8244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:14.526795 8244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:14.529156 8244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:14.529451 8244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:14.530046 8244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:07:14.530117 8244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:14.530165 8244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:07:14.530192 8244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 00:07:14 dist-test-slave-grqt krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914034, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 00:07:14.537387 8244 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42519
I20260620 00:07:14.537570 8374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42519 every 8 connection(s)
I20260620 00:07:14.537814 8244 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:07:14.540982 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:14.530910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37231 (local address 127.0.98.62:38645)
0620 00:07:14.530965 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:14.530967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:14.531127 (+ 160us) server_negotiation.cc:407] Connection header received
0620 00:07:14.531789 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:14.531790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:14.531804 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:14.531881 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:14.532815 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.533687 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.534737 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.534937 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.535031 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:14.538018 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:14.538021 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:14.538022 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:14.538031 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:14.539586 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:14.540005 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:14.540005 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:14.540006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:14.540058 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:14.540317 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:14.540317 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:14.540318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:14.540430 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:14.540514 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:14.540842 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 00:07:14.540915 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 00:07:14.541121 8268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:14.530741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38645 (local address 127.0.98.3:37231)
0620 00:07:14.530982 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:14.531032 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:14.531643 (+ 611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:14.531973 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:14.531978 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:14.532375 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:14.532710 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:14.532717 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.533797 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.533798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:14.534632 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:14.534634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.535052 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.535053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:14.535190 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:14.536186 (+ 996us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:14.536195 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:14.537871 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:14.539666 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:14.539672 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:14.539674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:14.539932 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:14.540152 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:14.540153 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:14.540153 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:14.540260 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:14.540536 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:14.540541 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:14.540676 (+ 135us) client_negotiation.cc:769] Sending connection context
0620 00:07:14.540775 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 00:07:14.540908 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":87,"threads_started":1}
I20260620 00:07:14.541718 8375 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38645
I20260620 00:07:14.541833 8375 heartbeater.cc:464] Registering TS with master...
I20260620 00:07:14.542027 8375 heartbeater.cc:511] Master 127.0.98.62:38645 requested a full tablet report, sending...
I20260620 00:07:14.542430 7891 ts_manager.cc:194] Registered new tserver with Master: d9fc1f19fe2a4e4f93eddca1b80f3fb4 (127.0.98.3:42519)
I20260620 00:07:14.542825 7891 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37231
I20260620 00:07:14.546911 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 8244
I20260620 00:07:14.547024 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:07:14.547267 392 external_mini_cluster.cc:1428] Setting key ab9395ab3046507df6b036629303e92c
I20260620 00:07:14.548187 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:07:14.549069 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:14.542908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52901 (local address 127.0.98.62:38645)
0620 00:07:14.542998 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:14.543000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:14.543050 (+ 50us) server_negotiation.cc:407] Connection header received
0620 00:07:14.543131 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:14.543132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:14.543141 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:14.543208 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:14.543617 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.544070 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.544750 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.544924 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.544974 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:14.545939 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:14.545940 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:14.545941 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:14.545947 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:14.547778 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:14.548238 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:14.548238 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:14.548239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:14.548323 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:14.548518 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:14.548519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:14.548519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:14.548639 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:14.548716 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:14.548890 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 00:07:14.548973 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 00:07:14.549088 8381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:14.542862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38645 (local address 127.0.98.3:52901)
0620 00:07:14.542972 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:14.542984 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:14.543064 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:14.543216 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:14.543217 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:14.543442 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:14.543498 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:14.543500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.544158 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.544159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:14.544676 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:14.544678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.544987 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.544988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:14.545034 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:14.545558 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:14.545563 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:14.545862 (+ 299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:14.547897 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:14.547898 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:14.547898 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:14.548147 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:14.548381 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:14.548383 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:14.548383 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:14.548449 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:14.548709 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:14.548710 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:14.548763 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 00:07:14.548865 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 00:07:14.548976 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":62,"threads_started":1}
Jun 20 00:07:14 dist-test-slave-grqt krb5kdc[7847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914034, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 00:07:14 dist-test-slave-grqt krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:14 dist-test-slave-grqt krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 00:07:14.572649 7959 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:14.566316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35544 (local address 127.0.98.62:38645)
0620 00:07:14.566382 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:14.566384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:14.566393 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:07:14.566419 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:14.566420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:14.566428 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:14.566481 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:14.567131 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.567714 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.568603 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.568798 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.568865 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:14.572344 (+ 3479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35544: BlockingRecv error: recv got EOF from 127.0.0.1:35544 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 20 00:07:14 dist-test-slave-grqt krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914034, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 00:07:14.581073 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:14.573323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35558 (local address 127.0.98.62:38645)
0620 00:07:14.573435 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:14.573437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:14.573471 (+ 34us) server_negotiation.cc:407] Connection header received
0620 00:07:14.573533 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:14.573533 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:14.573542 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:14.573598 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:14.574195 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.574699 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.575466 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.575656 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.575714 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:14.578261 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:14.578263 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:14.578263 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:14.578270 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:14.579725 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:14.580089 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:14.580090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:14.580090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:14.580230 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:14.580531 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:14.580532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:14.580532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:14.580680 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:14.580778 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:14.580911 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 00:07:14.580997 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 00:07:14.583451 7891 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:07:14.585258 7891 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:07:14.592536 8398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:14.588024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34395 (local address 127.0.0.1:47518)
0620 00:07:14.588877 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:14.588887 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:14.588931 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:14.589261 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:14.589263 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:14.589265 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:07:14.589302 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:14.589304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.590514 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.590515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:14.591436 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:14.591439 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.592293 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.592294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:14.592349 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:14.592352 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:07:14.592400 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 00:07:14.592433 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"mutex_wait_us":2,"thread_start_us":92,"threads_started":1}
I20260620 00:07:14.592649 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:14.588115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35558 (local address 127.0.98.1:35827)
0620 00:07:14.588626 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:14.588630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:14.588665 (+ 35us) server_negotiation.cc:407] Connection header received
0620 00:07:14.588704 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:14.588706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:14.588720 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:14.588798 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:07:14.588992 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.589908 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.591456 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.591988 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.592094 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:14.592198 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:14.592500 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 00:07:14.592542 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":76,"threads_started":1}
I20260620 00:07:14.592636 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:14.588024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42519 (local address 127.0.0.1:44294)
0620 00:07:14.588450 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:14.588460 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:14.588510 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:14.588702 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:14.588705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:14.588707 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:07:14.588790 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:14.588792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.590306 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.590306 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:14.591359 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:14.591361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.592387 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.592388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:14.592464 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:14.592468 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:07:14.592516 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 00:07:14.592563 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":90,"threads_started":1}
I20260620 00:07:14.592454 8397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:14.588017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35827 (local address 127.0.0.1:35558)
0620 00:07:14.588287 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:14.588332 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:14.588407 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:14.588796 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:14.588797 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:14.588799 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:07:14.588897 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:14.588900 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.590008 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.590009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:14.591359 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:14.591361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.592099 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.592100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:14.592182 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:14.592198 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 00:07:14.592296 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 00:07:14.592352 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":1,"thread_start_us":74,"threads_started":1}
I20260620 00:07:14.593000 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:14.588271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44294 (local address 127.0.98.3:42519)
0620 00:07:14.588452 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:14.588457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:14.588502 (+ 45us) server_negotiation.cc:407] Connection header received
0620 00:07:14.588576 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:14.588578 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:14.588591 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:14.588661 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:07:14.588880 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.590182 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.591466 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.592212 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.592277 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:14.592353 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:14.592822 (+ 469us) server_negotiation.cc:299] Negotiation successful
0620 00:07:14.592871 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":91,"threads_started":1}
I20260620 00:07:14.593161 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:14.588302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47518 (local address 127.0.98.2:34395)
0620 00:07:14.589053 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:14.589058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:14.589068 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:07:14.589105 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:14.589107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:14.589122 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:14.589241 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:07:14.589412 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.590430 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.591518 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.592202 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.592278 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:14.592349 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:14.592424 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 00:07:14.592466 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":713,"thread_start_us":106,"threads_started":1}
I20260620 00:07:14.593580 8309 tablet_service.cc:1467] Processing CreateTablet for tablet 5a5d77b16f8d414cb0a795f93b70d3a5 (DEFAULT_TABLE table=test-table [id=f5993d240ded44cda11fd9084f2c78fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:07:14.593580 8029 tablet_service.cc:1467] Processing CreateTablet for tablet 5a5d77b16f8d414cb0a795f93b70d3a5 (DEFAULT_TABLE table=test-table [id=f5993d240ded44cda11fd9084f2c78fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:07:14.593758 8169 tablet_service.cc:1467] Processing CreateTablet for tablet 5a5d77b16f8d414cb0a795f93b70d3a5 (DEFAULT_TABLE table=test-table [id=f5993d240ded44cda11fd9084f2c78fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:07:14.594028 8169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a5d77b16f8d414cb0a795f93b70d3a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:14.594049 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a5d77b16f8d414cb0a795f93b70d3a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:14.594049 8029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a5d77b16f8d414cb0a795f93b70d3a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:14.597210 8403 tablet_bootstrap.cc:492] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 69ca88ea258443a39dad68029d288f95: Bootstrap starting.
I20260620 00:07:14.597886 8403 tablet_bootstrap.cc:654] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 69ca88ea258443a39dad68029d288f95: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:14.597994 8404 tablet_bootstrap.cc:492] T 5a5d77b16f8d414cb0a795f93b70d3a5 P d9fc1f19fe2a4e4f93eddca1b80f3fb4: Bootstrap starting.
I20260620 00:07:14.598237 8403 log.cc:824] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 69ca88ea258443a39dad68029d288f95: Log is configured to *not* fsync() on all Append() calls
I20260620 00:07:14.598618 8404 tablet_bootstrap.cc:654] T 5a5d77b16f8d414cb0a795f93b70d3a5 P d9fc1f19fe2a4e4f93eddca1b80f3fb4: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:14.598649 8405 tablet_bootstrap.cc:492] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 6f34094d8e684686b4004cf910f8f16a: Bootstrap starting.
I20260620 00:07:14.598889 8404 log.cc:824] T 5a5d77b16f8d414cb0a795f93b70d3a5 P d9fc1f19fe2a4e4f93eddca1b80f3fb4: Log is configured to *not* fsync() on all Append() calls
I20260620 00:07:14.599232 8403 tablet_bootstrap.cc:492] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 69ca88ea258443a39dad68029d288f95: No bootstrap required, opened a new log
I20260620 00:07:14.599283 8405 tablet_bootstrap.cc:654] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 6f34094d8e684686b4004cf910f8f16a: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:14.599323 8403 ts_tablet_manager.cc:1397] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 69ca88ea258443a39dad68029d288f95: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:07:14.599620 8404 tablet_bootstrap.cc:492] T 5a5d77b16f8d414cb0a795f93b70d3a5 P d9fc1f19fe2a4e4f93eddca1b80f3fb4: No bootstrap required, opened a new log
I20260620 00:07:14.599612 8405 log.cc:824] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 6f34094d8e684686b4004cf910f8f16a: Log is configured to *not* fsync() on all Append() calls
I20260620 00:07:14.599699 8404 ts_tablet_manager.cc:1397] T 5a5d77b16f8d414cb0a795f93b70d3a5 P d9fc1f19fe2a4e4f93eddca1b80f3fb4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:07:14.600383 8405 tablet_bootstrap.cc:492] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 6f34094d8e684686b4004cf910f8f16a: No bootstrap required, opened a new log
I20260620 00:07:14.600456 8405 ts_tablet_manager.cc:1397] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 6f34094d8e684686b4004cf910f8f16a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:07:14.601320 8403 raft_consensus.cc:348] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 69ca88ea258443a39dad68029d288f95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } }
I20260620 00:07:14.601545 8403 raft_consensus.cc:374] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 69ca88ea258443a39dad68029d288f95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:14.601593 8403 raft_consensus.cc:729] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 69ca88ea258443a39dad68029d288f95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ca88ea258443a39dad68029d288f95, State: Initialized, Role: FOLLOWER
I20260620 00:07:14.601711 8403 consensus_queue.cc:260] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 69ca88ea258443a39dad68029d288f95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } }
I20260620 00:07:14.602010 8235 heartbeater.cc:503] Master 127.0.98.62:38645 was elected leader, sending a full tablet report...
I20260620 00:07:14.601977 8404 raft_consensus.cc:348] T 5a5d77b16f8d414cb0a795f93b70d3a5 P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } }
I20260620 00:07:14.602139 8404 raft_consensus.cc:374] T 5a5d77b16f8d414cb0a795f93b70d3a5 P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:14.602181 8404 raft_consensus.cc:729] T 5a5d77b16f8d414cb0a795f93b70d3a5 P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9fc1f19fe2a4e4f93eddca1b80f3fb4, State: Initialized, Role: FOLLOWER
I20260620 00:07:14.602299 8404 consensus_queue.cc:260] T 5a5d77b16f8d414cb0a795f93b70d3a5 P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } }
I20260620 00:07:14.602582 8375 heartbeater.cc:503] Master 127.0.98.62:38645 was elected leader, sending a full tablet report...
I20260620 00:07:14.602614 8405 raft_consensus.cc:348] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 6f34094d8e684686b4004cf910f8f16a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } }
I20260620 00:07:14.602763 8405 raft_consensus.cc:374] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 6f34094d8e684686b4004cf910f8f16a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:14.602803 8405 raft_consensus.cc:729] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 6f34094d8e684686b4004cf910f8f16a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f34094d8e684686b4004cf910f8f16a, State: Initialized, Role: FOLLOWER
I20260620 00:07:14.602932 8403 ts_tablet_manager.cc:1428] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 69ca88ea258443a39dad68029d288f95: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260620 00:07:14.602916 8405 consensus_queue.cc:260] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 6f34094d8e684686b4004cf910f8f16a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } }
I20260620 00:07:14.602977 8404 ts_tablet_manager.cc:1428] T 5a5d77b16f8d414cb0a795f93b70d3a5 P d9fc1f19fe2a4e4f93eddca1b80f3fb4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:07:14.603178 8405 ts_tablet_manager.cc:1428] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 6f34094d8e684686b4004cf910f8f16a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:07:14.603336 8095 heartbeater.cc:503] Master 127.0.98.62:38645 was elected leader, sending a full tablet report...
W20260620 00:07:14.616492 8236 tablet.cc:2369] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 69ca88ea258443a39dad68029d288f95: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 00:07:14.697860 8096 tablet.cc:2369] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 6f34094d8e684686b4004cf910f8f16a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:07:14.706947 8411 raft_consensus.cc:482] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 6f34094d8e684686b4004cf910f8f16a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:07:14.707126 8411 raft_consensus.cc:504] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 6f34094d8e684686b4004cf910f8f16a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } }
I20260620 00:07:14.707568 8411 leader_election.cc:288] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 6f34094d8e684686b4004cf910f8f16a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9fc1f19fe2a4e4f93eddca1b80f3fb4 (127.0.98.3:42519), 69ca88ea258443a39dad68029d288f95 (127.0.98.2:34395)
I20260620 00:07:14.710717 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:14.707872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35327 (local address 127.0.98.3:42519)
0620 00:07:14.707933 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:14.707935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:14.708055 (+ 120us) server_negotiation.cc:407] Connection header received
0620 00:07:14.708132 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:14.708133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:14.708145 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:14.708226 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:07:14.708386 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.708860 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.709984 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.710344 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.710432 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:14.710442 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:14.710599 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 00:07:14.710638 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 00:07:14.710749 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:14.707894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41449 (local address 127.0.98.2:34395)
0620 00:07:14.707944 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:14.707947 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:14.708067 (+ 120us) server_negotiation.cc:407] Connection header received
0620 00:07:14.708140 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:14.708141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:14.708152 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:14.708229 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:07:14.708397 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.708908 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.709987 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.710352 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.710439 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:14.710449 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:14.710626 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 00:07:14.710675 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 00:07:14.710770 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:14.707818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34395 (local address 127.0.98.1:41449)
0620 00:07:14.707990 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:14.707999 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:14.708083 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:14.708232 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:14.708233 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:14.708234 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:07:14.708316 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:14.708318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.708992 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.708993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:14.709893 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:14.709895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.710438 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.710439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:14.710523 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:14.710526 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:07:14.710619 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 00:07:14.710666 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":27,"thread_start_us":83,"threads_started":1}
I20260620 00:07:14.711053 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:14.707785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42519 (local address 127.0.98.1:35327)
0620 00:07:14.707973 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:14.707984 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:14.708074 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:14.708231 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:14.708232 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:14.708234 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:07:14.708313 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:14.708315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.708956 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.708957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:14.709880 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:14.709882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.710431 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.710432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:14.710507 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:14.710511 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:07:14.710590 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 00:07:14.710636 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":59,"threads_started":1}
I20260620 00:07:14.711378 8189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a5d77b16f8d414cb0a795f93b70d3a5" candidate_uuid: "6f34094d8e684686b4004cf910f8f16a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ca88ea258443a39dad68029d288f95" is_pre_election: true
I20260620 00:07:14.711488 8329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a5d77b16f8d414cb0a795f93b70d3a5" candidate_uuid: "6f34094d8e684686b4004cf910f8f16a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" is_pre_election: true
I20260620 00:07:14.711565 8189 raft_consensus.cc:2454] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 69ca88ea258443a39dad68029d288f95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f34094d8e684686b4004cf910f8f16a in term 0.
I20260620 00:07:14.711658 8329 raft_consensus.cc:2454] T 5a5d77b16f8d414cb0a795f93b70d3a5 P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f34094d8e684686b4004cf910f8f16a in term 0.
I20260620 00:07:14.711819 7983 leader_election.cc:302] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 6f34094d8e684686b4004cf910f8f16a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ca88ea258443a39dad68029d288f95, 6f34094d8e684686b4004cf910f8f16a; no voters:
I20260620 00:07:14.711931 8411 raft_consensus.cc:2781] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 6f34094d8e684686b4004cf910f8f16a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:07:14.712018 8411 raft_consensus.cc:482] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 6f34094d8e684686b4004cf910f8f16a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:07:14.712056 8411 raft_consensus.cc:3037] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 6f34094d8e684686b4004cf910f8f16a [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:14.712996 8411 raft_consensus.cc:504] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 6f34094d8e684686b4004cf910f8f16a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } }
I20260620 00:07:14.713220 8411 leader_election.cc:288] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 6f34094d8e684686b4004cf910f8f16a [CANDIDATE]: Term 1 election: Requested vote from peers d9fc1f19fe2a4e4f93eddca1b80f3fb4 (127.0.98.3:42519), 69ca88ea258443a39dad68029d288f95 (127.0.98.2:34395)
I20260620 00:07:14.713332 8329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a5d77b16f8d414cb0a795f93b70d3a5" candidate_uuid: "6f34094d8e684686b4004cf910f8f16a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4"
I20260620 00:07:14.713428 8329 raft_consensus.cc:3037] T 5a5d77b16f8d414cb0a795f93b70d3a5 P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:14.713436 8189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a5d77b16f8d414cb0a795f93b70d3a5" candidate_uuid: "6f34094d8e684686b4004cf910f8f16a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ca88ea258443a39dad68029d288f95"
I20260620 00:07:14.713505 8189 raft_consensus.cc:3037] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 69ca88ea258443a39dad68029d288f95 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:14.714251 8329 raft_consensus.cc:2454] T 5a5d77b16f8d414cb0a795f93b70d3a5 P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f34094d8e684686b4004cf910f8f16a in term 1.
I20260620 00:07:14.714308 8189 raft_consensus.cc:2454] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 69ca88ea258443a39dad68029d288f95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f34094d8e684686b4004cf910f8f16a in term 1.
I20260620 00:07:14.714488 7982 leader_election.cc:302] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 6f34094d8e684686b4004cf910f8f16a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f34094d8e684686b4004cf910f8f16a, d9fc1f19fe2a4e4f93eddca1b80f3fb4; no voters:
I20260620 00:07:14.714589 8411 raft_consensus.cc:2781] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 6f34094d8e684686b4004cf910f8f16a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:07:14.714722 8411 raft_consensus.cc:686] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 6f34094d8e684686b4004cf910f8f16a [term 1 LEADER]: Becoming Leader. State: Replica: 6f34094d8e684686b4004cf910f8f16a, State: Running, Role: LEADER
I20260620 00:07:14.714841 8411 consensus_queue.cc:237] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 6f34094d8e684686b4004cf910f8f16a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } }
I20260620 00:07:14.715651 7891 catalog_manager.cc:5495] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 6f34094d8e684686b4004cf910f8f16a reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f34094d8e684686b4004cf910f8f16a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6f34094d8e684686b4004cf910f8f16a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } health_report { overall_health: UNKNOWN } } }
I20260620 00:07:14.739444 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:14.736675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35574 (local address 127.0.98.1:35827)
0620 00:07:14.736768 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:14.736770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:14.736779 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:07:14.736817 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:14.736818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:14.736837 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:14.736907 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:07:14.737076 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.737651 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.738459 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.738720 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.738816 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:14.739038 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:07:14.739146 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:07:14.739211 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:14.739307 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 00:07:14.739328 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 00:07:14.741520 8329 raft_consensus.cc:1261] T 5a5d77b16f8d414cb0a795f93b70d3a5 P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 1 FOLLOWER]: Refusing update from remote peer 6f34094d8e684686b4004cf910f8f16a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:07:14.741544 8189 raft_consensus.cc:1261] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 69ca88ea258443a39dad68029d288f95 [term 1 FOLLOWER]: Refusing update from remote peer 6f34094d8e684686b4004cf910f8f16a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:07:14.741921 8411 consensus_queue.cc:1035] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 6f34094d8e684686b4004cf910f8f16a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:07:14.742079 8414 consensus_queue.cc:1035] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 6f34094d8e684686b4004cf910f8f16a [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:07:14.744526 8417 mvcc.cc:205] Tried to move back new op lower bound from 7298719886298046464 to 7298719886192660480. Current Snapshot: MvccSnapshot[applied={T|T < 7298719886298046464}]
I20260620 00:07:14.745299 8419 mvcc.cc:205] Tried to move back new op lower bound from 7298719886298046464 to 7298719886192660480. Current Snapshot: MvccSnapshot[applied={T|T < 7298719886298046464 or (T in {7298719886298046464})}]
I20260620 00:07:14.745445 8418 mvcc.cc:205] Tried to move back new op lower bound from 7298719886298046464 to 7298719886192660480. Current Snapshot: MvccSnapshot[applied={T|T < 7298719886298046464 or (T in {7298719886298046464})}]
I20260620 00:07:14.747036 7892 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:07:14.747143 7892 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:07:14.747929 7892 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05cccc5116e84b959eee49c8c6ca05a4: Sending DeleteTablet for 3 replicas of tablet 5a5d77b16f8d414cb0a795f93b70d3a5
I20260620 00:07:14.748320 8029 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a5d77b16f8d414cb0a795f93b70d3a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:07:14 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35558
I20260620 00:07:14.748448 8169 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a5d77b16f8d414cb0a795f93b70d3a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:07:14 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47518
I20260620 00:07:14.748591 8427 tablet_replica.cc:330] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 6f34094d8e684686b4004cf910f8f16a: stopping tablet replica
I20260620 00:07:14.748633 8309 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a5d77b16f8d414cb0a795f93b70d3a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:07:14 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44294
I20260620 00:07:14.748695 8427 raft_consensus.cc:2229] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 6f34094d8e684686b4004cf910f8f16a [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:07:14.748785 8427 raft_consensus.cc:2258] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 6f34094d8e684686b4004cf910f8f16a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:07:14.749071 8429 tablet_replica.cc:330] T 5a5d77b16f8d414cb0a795f93b70d3a5 P d9fc1f19fe2a4e4f93eddca1b80f3fb4: stopping tablet replica
I20260620 00:07:14.749068 7892 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:07:14.749233 7892 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:07:14.749246 8427 ts_tablet_manager.cc:1905] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 6f34094d8e684686b4004cf910f8f16a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:07:14.749454 8429 raft_consensus.cc:2229] T 5a5d77b16f8d414cb0a795f93b70d3a5 P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:07:14.749645 8429 raft_consensus.cc:2258] T 5a5d77b16f8d414cb0a795f93b70d3a5 P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:07:14.750146 8429 ts_tablet_manager.cc:1905] T 5a5d77b16f8d414cb0a795f93b70d3a5 P d9fc1f19fe2a4e4f93eddca1b80f3fb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:07:14.750830 8029 tablet_service.cc:1467] Processing CreateTablet for tablet 2334f24224a84dd3849cff0f57142b0e (DEFAULT_TABLE table=test-table [id=14439cf54eef4227bd5adaa2b224afba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:07:14.751045 8029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2334f24224a84dd3849cff0f57142b0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:14.751168 8309 tablet_service.cc:1467] Processing CreateTablet for tablet 2334f24224a84dd3849cff0f57142b0e (DEFAULT_TABLE table=test-table [id=14439cf54eef4227bd5adaa2b224afba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:07:14.751332 8169 tablet_service.cc:1467] Processing CreateTablet for tablet 2334f24224a84dd3849cff0f57142b0e (DEFAULT_TABLE table=test-table [id=14439cf54eef4227bd5adaa2b224afba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:07:14.751395 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2334f24224a84dd3849cff0f57142b0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:14.751434 8169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2334f24224a84dd3849cff0f57142b0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:14.752244 8405 tablet_bootstrap.cc:492] T 2334f24224a84dd3849cff0f57142b0e P 6f34094d8e684686b4004cf910f8f16a: Bootstrap starting.
I20260620 00:07:14.752566 8404 tablet_bootstrap.cc:492] T 2334f24224a84dd3849cff0f57142b0e P d9fc1f19fe2a4e4f93eddca1b80f3fb4: Bootstrap starting.
I20260620 00:07:14.752641 8405 tablet_bootstrap.cc:654] T 2334f24224a84dd3849cff0f57142b0e P 6f34094d8e684686b4004cf910f8f16a: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:14.752763 8403 tablet_bootstrap.cc:492] T 2334f24224a84dd3849cff0f57142b0e P 69ca88ea258443a39dad68029d288f95: Bootstrap starting.
I20260620 00:07:14.752983 8404 tablet_bootstrap.cc:654] T 2334f24224a84dd3849cff0f57142b0e P d9fc1f19fe2a4e4f93eddca1b80f3fb4: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:14.751047 8427 ts_tablet_manager.cc:1918] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 6f34094d8e684686b4004cf910f8f16a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:07:14.753103 8428 tablet_replica.cc:330] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 69ca88ea258443a39dad68029d288f95: stopping tablet replica
I20260620 00:07:14.753152 8427 log.cc:1196] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 6f34094d8e684686b4004cf910f8f16a: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-0/wal/wals/5a5d77b16f8d414cb0a795f93b70d3a5
I20260620 00:07:14.753175 8403 tablet_bootstrap.cc:654] T 2334f24224a84dd3849cff0f57142b0e P 69ca88ea258443a39dad68029d288f95: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:14.753217 8428 raft_consensus.cc:2229] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 69ca88ea258443a39dad68029d288f95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:07:14.753450 8427 ts_tablet_manager.cc:1939] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 6f34094d8e684686b4004cf910f8f16a: Deleting consensus metadata
I20260620 00:07:14.753711 8405 tablet_bootstrap.cc:492] T 2334f24224a84dd3849cff0f57142b0e P 6f34094d8e684686b4004cf910f8f16a: No bootstrap required, opened a new log
I20260620 00:07:14.753765 8405 ts_tablet_manager.cc:1397] T 2334f24224a84dd3849cff0f57142b0e P 6f34094d8e684686b4004cf910f8f16a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:07:14.753830 7877 catalog_manager.cc:4841] TS 6f34094d8e684686b4004cf910f8f16a (127.0.98.1:35827): tablet 5a5d77b16f8d414cb0a795f93b70d3a5 (table test-table [id=f5993d240ded44cda11fd9084f2c78fe]) successfully deleted
I20260620 00:07:14.753922 8405 raft_consensus.cc:348] T 2334f24224a84dd3849cff0f57142b0e P 6f34094d8e684686b4004cf910f8f16a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } }
I20260620 00:07:14.754007 8405 raft_consensus.cc:374] T 2334f24224a84dd3849cff0f57142b0e P 6f34094d8e684686b4004cf910f8f16a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:14.754036 8405 raft_consensus.cc:729] T 2334f24224a84dd3849cff0f57142b0e P 6f34094d8e684686b4004cf910f8f16a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f34094d8e684686b4004cf910f8f16a, State: Initialized, Role: FOLLOWER
I20260620 00:07:14.754086 8405 consensus_queue.cc:260] T 2334f24224a84dd3849cff0f57142b0e P 6f34094d8e684686b4004cf910f8f16a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } }
I20260620 00:07:14.754175 8405 ts_tablet_manager.cc:1428] T 2334f24224a84dd3849cff0f57142b0e P 6f34094d8e684686b4004cf910f8f16a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:14.753345 8428 raft_consensus.cc:2258] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 69ca88ea258443a39dad68029d288f95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:07:14.754877 8429 ts_tablet_manager.cc:1918] T 5a5d77b16f8d414cb0a795f93b70d3a5 P d9fc1f19fe2a4e4f93eddca1b80f3fb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:07:14.754997 8429 log.cc:1196] T 5a5d77b16f8d414cb0a795f93b70d3a5 P d9fc1f19fe2a4e4f93eddca1b80f3fb4: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-2/wal/wals/5a5d77b16f8d414cb0a795f93b70d3a5
I20260620 00:07:14.755112 8428 ts_tablet_manager.cc:1905] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 69ca88ea258443a39dad68029d288f95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:07:14.755360 8429 ts_tablet_manager.cc:1939] T 5a5d77b16f8d414cb0a795f93b70d3a5 P d9fc1f19fe2a4e4f93eddca1b80f3fb4: Deleting consensus metadata
I20260620 00:07:14.755815 7878 catalog_manager.cc:4841] TS d9fc1f19fe2a4e4f93eddca1b80f3fb4 (127.0.98.3:42519): tablet 5a5d77b16f8d414cb0a795f93b70d3a5 (table test-table [id=f5993d240ded44cda11fd9084f2c78fe]) successfully deleted
I20260620 00:07:14.756551 8428 ts_tablet_manager.cc:1918] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 69ca88ea258443a39dad68029d288f95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:07:14.756623 8428 log.cc:1196] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 69ca88ea258443a39dad68029d288f95: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-1/wal/wals/5a5d77b16f8d414cb0a795f93b70d3a5
I20260620 00:07:14.756848 8428 ts_tablet_manager.cc:1939] T 5a5d77b16f8d414cb0a795f93b70d3a5 P 69ca88ea258443a39dad68029d288f95: Deleting consensus metadata
I20260620 00:07:14.757266 7879 catalog_manager.cc:4841] TS 69ca88ea258443a39dad68029d288f95 (127.0.98.2:34395): tablet 5a5d77b16f8d414cb0a795f93b70d3a5 (table test-table [id=f5993d240ded44cda11fd9084f2c78fe]) successfully deleted
I20260620 00:07:14.757503 8403 tablet_bootstrap.cc:492] T 2334f24224a84dd3849cff0f57142b0e P 69ca88ea258443a39dad68029d288f95: No bootstrap required, opened a new log
I20260620 00:07:14.757535 8403 ts_tablet_manager.cc:1397] T 2334f24224a84dd3849cff0f57142b0e P 69ca88ea258443a39dad68029d288f95: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260620 00:07:14.757647 8403 raft_consensus.cc:348] T 2334f24224a84dd3849cff0f57142b0e P 69ca88ea258443a39dad68029d288f95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } }
I20260620 00:07:14.757702 8403 raft_consensus.cc:374] T 2334f24224a84dd3849cff0f57142b0e P 69ca88ea258443a39dad68029d288f95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:14.757716 8403 raft_consensus.cc:729] T 2334f24224a84dd3849cff0f57142b0e P 69ca88ea258443a39dad68029d288f95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ca88ea258443a39dad68029d288f95, State: Initialized, Role: FOLLOWER
I20260620 00:07:14.757753 8403 consensus_queue.cc:260] T 2334f24224a84dd3849cff0f57142b0e P 69ca88ea258443a39dad68029d288f95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } }
I20260620 00:07:14.757854 8403 ts_tablet_manager.cc:1428] T 2334f24224a84dd3849cff0f57142b0e P 69ca88ea258443a39dad68029d288f95: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:14.761909 8404 tablet_bootstrap.cc:492] T 2334f24224a84dd3849cff0f57142b0e P d9fc1f19fe2a4e4f93eddca1b80f3fb4: No bootstrap required, opened a new log
I20260620 00:07:14.761997 8404 ts_tablet_manager.cc:1397] T 2334f24224a84dd3849cff0f57142b0e P d9fc1f19fe2a4e4f93eddca1b80f3fb4: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260620 00:07:14.762184 8404 raft_consensus.cc:348] T 2334f24224a84dd3849cff0f57142b0e P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } }
I20260620 00:07:14.762281 8404 raft_consensus.cc:374] T 2334f24224a84dd3849cff0f57142b0e P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:14.762329 8404 raft_consensus.cc:729] T 2334f24224a84dd3849cff0f57142b0e P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9fc1f19fe2a4e4f93eddca1b80f3fb4, State: Initialized, Role: FOLLOWER
I20260620 00:07:14.762387 8404 consensus_queue.cc:260] T 2334f24224a84dd3849cff0f57142b0e P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } }
I20260620 00:07:14.762516 8404 ts_tablet_manager.cc:1428] T 2334f24224a84dd3849cff0f57142b0e P d9fc1f19fe2a4e4f93eddca1b80f3fb4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 00:07:14.777606 8409 raft_consensus.cc:482] T 2334f24224a84dd3849cff0f57142b0e P 69ca88ea258443a39dad68029d288f95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:07:14.777698 8409 raft_consensus.cc:504] T 2334f24224a84dd3849cff0f57142b0e P 69ca88ea258443a39dad68029d288f95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } }
I20260620 00:07:14.778062 8409 leader_election.cc:288] T 2334f24224a84dd3849cff0f57142b0e P 69ca88ea258443a39dad68029d288f95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f34094d8e684686b4004cf910f8f16a (127.0.98.1:35827), d9fc1f19fe2a4e4f93eddca1b80f3fb4 (127.0.98.3:42519)
I20260620 00:07:14.782195 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:14.778297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43721 (local address 127.0.98.1:35827)
0620 00:07:14.778427 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:14.778429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:14.778636 (+ 207us) server_negotiation.cc:407] Connection header received
0620 00:07:14.778697 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:14.778698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:14.778707 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:14.778753 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:07:14.778983 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.779453 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.781044 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.781698 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.781787 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:14.781799 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:14.782070 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 00:07:14.782114 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260620 00:07:14.782231 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:14.778229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35827 (local address 127.0.98.2:43721)
0620 00:07:14.778573 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:14.778593 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:14.778645 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:14.778807 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:14.778808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:14.778810 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:07:14.778888 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:14.778889 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.779542 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.779544 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:14.780931 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:14.780934 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.781829 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.781831 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:14.781957 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:14.781963 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 00:07:14.782059 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 00:07:14.782106 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":272,"thread_start_us":75,"threads_started":1}
I20260620 00:07:14.782594 8049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2334f24224a84dd3849cff0f57142b0e" candidate_uuid: "69ca88ea258443a39dad68029d288f95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f34094d8e684686b4004cf910f8f16a" is_pre_election: true
I20260620 00:07:14.782748 8049 raft_consensus.cc:2454] T 2334f24224a84dd3849cff0f57142b0e P 6f34094d8e684686b4004cf910f8f16a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ca88ea258443a39dad68029d288f95 in term 0.
I20260620 00:07:14.783071 8128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:14.778229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42519 (local address 127.0.98.2:58519)
0620 00:07:14.778289 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:14.778300 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:14.778346 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:14.778574 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:14.778575 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:14.778578 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:07:14.778652 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:14.778654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.779602 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.779603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:14.780931 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:14.780934 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.782755 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.782756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:14.782853 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:14.782857 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:07:14.782944 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 00:07:14.783004 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":30}
I20260620 00:07:14.783070 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:14.778396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58519 (local address 127.0.98.3:42519)
0620 00:07:14.778441 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:14.778443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:14.778451 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:07:14.778479 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:14.778480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:14.778490 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:14.778548 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:07:14.778748 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.779518 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.781839 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.782653 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.782745 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:14.782756 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:14.782953 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 00:07:14.783000 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 00:07:14.783342 8329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2334f24224a84dd3849cff0f57142b0e" candidate_uuid: "69ca88ea258443a39dad68029d288f95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" is_pre_election: true
I20260620 00:07:14.783349 8121 leader_election.cc:302] T 2334f24224a84dd3849cff0f57142b0e P 69ca88ea258443a39dad68029d288f95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ca88ea258443a39dad68029d288f95, 6f34094d8e684686b4004cf910f8f16a; no voters:
I20260620 00:07:14.783425 8329 raft_consensus.cc:2454] T 2334f24224a84dd3849cff0f57142b0e P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ca88ea258443a39dad68029d288f95 in term 0.
I20260620 00:07:14.783480 8409 raft_consensus.cc:2781] T 2334f24224a84dd3849cff0f57142b0e P 69ca88ea258443a39dad68029d288f95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:07:14.783514 8409 raft_consensus.cc:482] T 2334f24224a84dd3849cff0f57142b0e P 69ca88ea258443a39dad68029d288f95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:07:14.783540 8409 raft_consensus.cc:3037] T 2334f24224a84dd3849cff0f57142b0e P 69ca88ea258443a39dad68029d288f95 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:14.784394 8409 raft_consensus.cc:504] T 2334f24224a84dd3849cff0f57142b0e P 69ca88ea258443a39dad68029d288f95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } }
I20260620 00:07:14.784546 8409 leader_election.cc:288] T 2334f24224a84dd3849cff0f57142b0e P 69ca88ea258443a39dad68029d288f95 [CANDIDATE]: Term 1 election: Requested vote from peers 6f34094d8e684686b4004cf910f8f16a (127.0.98.1:35827), d9fc1f19fe2a4e4f93eddca1b80f3fb4 (127.0.98.3:42519)
I20260620 00:07:14.784732 8049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2334f24224a84dd3849cff0f57142b0e" candidate_uuid: "69ca88ea258443a39dad68029d288f95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f34094d8e684686b4004cf910f8f16a"
I20260620 00:07:14.784811 8049 raft_consensus.cc:3037] T 2334f24224a84dd3849cff0f57142b0e P 6f34094d8e684686b4004cf910f8f16a [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:14.785595 8049 raft_consensus.cc:2454] T 2334f24224a84dd3849cff0f57142b0e P 6f34094d8e684686b4004cf910f8f16a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ca88ea258443a39dad68029d288f95 in term 1.
I20260620 00:07:14.785723 8329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2334f24224a84dd3849cff0f57142b0e" candidate_uuid: "69ca88ea258443a39dad68029d288f95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4"
I20260620 00:07:14.785770 8121 leader_election.cc:302] T 2334f24224a84dd3849cff0f57142b0e P 69ca88ea258443a39dad68029d288f95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ca88ea258443a39dad68029d288f95, 6f34094d8e684686b4004cf910f8f16a; no voters:
I20260620 00:07:14.785801 8329 raft_consensus.cc:3037] T 2334f24224a84dd3849cff0f57142b0e P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:14.785892 8409 raft_consensus.cc:2781] T 2334f24224a84dd3849cff0f57142b0e P 69ca88ea258443a39dad68029d288f95 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:07:14.786018 8409 raft_consensus.cc:686] T 2334f24224a84dd3849cff0f57142b0e P 69ca88ea258443a39dad68029d288f95 [term 1 LEADER]: Becoming Leader. State: Replica: 69ca88ea258443a39dad68029d288f95, State: Running, Role: LEADER
I20260620 00:07:14.786093 8409 consensus_queue.cc:237] T 2334f24224a84dd3849cff0f57142b0e P 69ca88ea258443a39dad68029d288f95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } }
I20260620 00:07:14.786617 8329 raft_consensus.cc:2454] T 2334f24224a84dd3849cff0f57142b0e P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ca88ea258443a39dad68029d288f95 in term 1.
I20260620 00:07:14.787050 7892 catalog_manager.cc:5495] T 2334f24224a84dd3849cff0f57142b0e P 69ca88ea258443a39dad68029d288f95 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69ca88ea258443a39dad68029d288f95 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "69ca88ea258443a39dad68029d288f95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } health_report { overall_health: UNKNOWN } } }
W20260620 00:07:14.788460 8376 tablet.cc:2369] T 2334f24224a84dd3849cff0f57142b0e P d9fc1f19fe2a4e4f93eddca1b80f3fb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 00:07:14 dist-test-slave-grqt krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914033, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 00:07:14.833545 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:14.826186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38645 (local address 127.0.0.1:35562)
0620 00:07:14.826353 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:14.826364 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:14.826458 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:14.826621 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:14.826622 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:14.826943 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:14.826987 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:14.826990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.827661 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.827662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:14.828177 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:14.828178 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.828558 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.828559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:14.828614 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:14.829164 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:14.829169 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:14.830559 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:14.832353 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:14.832361 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:14.832363 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:14.832573 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:14.832804 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:14.832805 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:14.832805 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:14.832864 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:14.833112 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:14.833120 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:14.833168 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 00:07:14.833248 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 00:07:14.833421 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":82,"threads_started":1}
I20260620 00:07:14.833549 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:14.826297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35562 (local address 127.0.98.62:38645)
0620 00:07:14.826345 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:14.826347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:14.826446 (+ 99us) server_negotiation.cc:407] Connection header received
0620 00:07:14.826525 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:14.826526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:14.826535 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:14.826614 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:14.827078 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.827568 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.828269 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:14.828453 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:14.828538 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:14.830658 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:14.830659 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:14.830659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:14.830666 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:14.832233 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:14.832666 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:14.832667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:14.832668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:14.832726 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:14.832929 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:14.832930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:14.832930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:14.833045 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:14.833123 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:14.833283 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 00:07:14.833472 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":119,"server-negotiator.queue_time_us":30}
I20260620 00:07:14.836166 7892 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35562:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 00:07:14.836370 7892 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:07:14.837847 8169 tablet_service.cc:1467] Processing CreateTablet for tablet f29ff8aba72e4e95a2d899d2dcb7aee6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5da56858cbf24f2eaefe3f6eaa530f89]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 00:07:14.837880 8309 tablet_service.cc:1467] Processing CreateTablet for tablet f29ff8aba72e4e95a2d899d2dcb7aee6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5da56858cbf24f2eaefe3f6eaa530f89]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 00:07:14.837922 8029 tablet_service.cc:1467] Processing CreateTablet for tablet f29ff8aba72e4e95a2d899d2dcb7aee6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5da56858cbf24f2eaefe3f6eaa530f89]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 00:07:14.838014 8169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f29ff8aba72e4e95a2d899d2dcb7aee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:14.838014 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f29ff8aba72e4e95a2d899d2dcb7aee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:14.838115 8029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f29ff8aba72e4e95a2d899d2dcb7aee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:14.839596 8404 tablet_bootstrap.cc:492] T f29ff8aba72e4e95a2d899d2dcb7aee6 P d9fc1f19fe2a4e4f93eddca1b80f3fb4: Bootstrap starting.
I20260620 00:07:14.839654 8405 tablet_bootstrap.cc:492] T f29ff8aba72e4e95a2d899d2dcb7aee6 P 6f34094d8e684686b4004cf910f8f16a: Bootstrap starting.
I20260620 00:07:14.839989 8404 tablet_bootstrap.cc:654] T f29ff8aba72e4e95a2d899d2dcb7aee6 P d9fc1f19fe2a4e4f93eddca1b80f3fb4: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:14.840041 8405 tablet_bootstrap.cc:654] T f29ff8aba72e4e95a2d899d2dcb7aee6 P 6f34094d8e684686b4004cf910f8f16a: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:14.840327 8403 tablet_bootstrap.cc:492] T f29ff8aba72e4e95a2d899d2dcb7aee6 P 69ca88ea258443a39dad68029d288f95: Bootstrap starting.
I20260620 00:07:14.840739 8403 tablet_bootstrap.cc:654] T f29ff8aba72e4e95a2d899d2dcb7aee6 P 69ca88ea258443a39dad68029d288f95: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:14.840987 8405 tablet_bootstrap.cc:492] T f29ff8aba72e4e95a2d899d2dcb7aee6 P 6f34094d8e684686b4004cf910f8f16a: No bootstrap required, opened a new log
I20260620 00:07:14.841033 8405 ts_tablet_manager.cc:1397] T f29ff8aba72e4e95a2d899d2dcb7aee6 P 6f34094d8e684686b4004cf910f8f16a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:14.841166 8405 raft_consensus.cc:348] T f29ff8aba72e4e95a2d899d2dcb7aee6 P 6f34094d8e684686b4004cf910f8f16a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } }
I20260620 00:07:14.841209 8404 tablet_bootstrap.cc:492] T f29ff8aba72e4e95a2d899d2dcb7aee6 P d9fc1f19fe2a4e4f93eddca1b80f3fb4: No bootstrap required, opened a new log
I20260620 00:07:14.841230 8405 raft_consensus.cc:374] T f29ff8aba72e4e95a2d899d2dcb7aee6 P 6f34094d8e684686b4004cf910f8f16a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:14.841250 8405 raft_consensus.cc:729] T f29ff8aba72e4e95a2d899d2dcb7aee6 P 6f34094d8e684686b4004cf910f8f16a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f34094d8e684686b4004cf910f8f16a, State: Initialized, Role: FOLLOWER
I20260620 00:07:14.841250 8404 ts_tablet_manager.cc:1397] T f29ff8aba72e4e95a2d899d2dcb7aee6 P d9fc1f19fe2a4e4f93eddca1b80f3fb4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:07:14.841301 8405 consensus_queue.cc:260] T f29ff8aba72e4e95a2d899d2dcb7aee6 P 6f34094d8e684686b4004cf910f8f16a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } }
I20260620 00:07:14.841423 8404 raft_consensus.cc:348] T f29ff8aba72e4e95a2d899d2dcb7aee6 P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } }
I20260620 00:07:14.841477 8405 ts_tablet_manager.cc:1428] T f29ff8aba72e4e95a2d899d2dcb7aee6 P 6f34094d8e684686b4004cf910f8f16a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:14.841501 8404 raft_consensus.cc:374] T f29ff8aba72e4e95a2d899d2dcb7aee6 P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:14.841454 8423 tablet_replica.cc:439] T f29ff8aba72e4e95a2d899d2dcb7aee6 P 6f34094d8e684686b4004cf910f8f16a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } } }
I20260620 00:07:14.841523 8404 raft_consensus.cc:729] T f29ff8aba72e4e95a2d899d2dcb7aee6 P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9fc1f19fe2a4e4f93eddca1b80f3fb4, State: Initialized, Role: FOLLOWER
I20260620 00:07:14.841542 8423 tablet_replica.cc:442] T f29ff8aba72e4e95a2d899d2dcb7aee6 P 6f34094d8e684686b4004cf910f8f16a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:07:14.841563 8404 consensus_queue.cc:260] T f29ff8aba72e4e95a2d899d2dcb7aee6 P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } }
I20260620 00:07:14.841650 8404 ts_tablet_manager.cc:1428] T f29ff8aba72e4e95a2d899d2dcb7aee6 P d9fc1f19fe2a4e4f93eddca1b80f3fb4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 00:07:14.841805 8410 tablet_replica.cc:439] T f29ff8aba72e4e95a2d899d2dcb7aee6 P d9fc1f19fe2a4e4f93eddca1b80f3fb4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } } }
I20260620 00:07:14.841892 8410 tablet_replica.cc:442] T f29ff8aba72e4e95a2d899d2dcb7aee6 P d9fc1f19fe2a4e4f93eddca1b80f3fb4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:07:14.842499 8403 tablet_bootstrap.cc:492] T f29ff8aba72e4e95a2d899d2dcb7aee6 P 69ca88ea258443a39dad68029d288f95: No bootstrap required, opened a new log
I20260620 00:07:14.842551 8403 ts_tablet_manager.cc:1397] T f29ff8aba72e4e95a2d899d2dcb7aee6 P 69ca88ea258443a39dad68029d288f95: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 00:07:14.842690 8403 raft_consensus.cc:348] T f29ff8aba72e4e95a2d899d2dcb7aee6 P 69ca88ea258443a39dad68029d288f95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } }
I20260620 00:07:14.842765 8403 raft_consensus.cc:374] T f29ff8aba72e4e95a2d899d2dcb7aee6 P 69ca88ea258443a39dad68029d288f95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:14.842779 8403 raft_consensus.cc:729] T f29ff8aba72e4e95a2d899d2dcb7aee6 P 69ca88ea258443a39dad68029d288f95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ca88ea258443a39dad68029d288f95, State: Initialized, Role: FOLLOWER
I20260620 00:07:14.842836 8403 consensus_queue.cc:260] T f29ff8aba72e4e95a2d899d2dcb7aee6 P 69ca88ea258443a39dad68029d288f95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } }
I20260620 00:07:14.842936 8403 ts_tablet_manager.cc:1428] T f29ff8aba72e4e95a2d899d2dcb7aee6 P 69ca88ea258443a39dad68029d288f95: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:14.842993 8409 tablet_replica.cc:439] T f29ff8aba72e4e95a2d899d2dcb7aee6 P 69ca88ea258443a39dad68029d288f95: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } } }
I20260620 00:07:14.843065 8409 tablet_replica.cc:442] T f29ff8aba72e4e95a2d899d2dcb7aee6 P 69ca88ea258443a39dad68029d288f95: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:07:15.065508 8410 raft_consensus.cc:482] T f29ff8aba72e4e95a2d899d2dcb7aee6 P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:07:15.065611 8410 raft_consensus.cc:504] T f29ff8aba72e4e95a2d899d2dcb7aee6 P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } }
I20260620 00:07:15.065953 8410 leader_election.cc:288] T f29ff8aba72e4e95a2d899d2dcb7aee6 P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f34094d8e684686b4004cf910f8f16a (127.0.98.1:35827), 69ca88ea258443a39dad68029d288f95 (127.0.98.2:34395)
I20260620 00:07:15.069038 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:15.066219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48199 (local address 127.0.98.1:35827)
0620 00:07:15.066284 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:15.066286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:15.066358 (+ 72us) server_negotiation.cc:407] Connection header received
0620 00:07:15.066460 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:15.066461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:15.066475 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:15.066537 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:07:15.066705 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:15.067191 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:15.068206 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:15.068578 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:15.068633 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:15.068641 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:15.068931 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 00:07:15.068969 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 00:07:15.069049 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:15.066251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53623 (local address 127.0.98.2:34395)
0620 00:07:15.066297 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:15.066299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:15.066400 (+ 101us) server_negotiation.cc:407] Connection header received
0620 00:07:15.066497 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:15.066498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:15.066504 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:15.066579 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:07:15.066758 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:15.067220 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:15.068243 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:15.068585 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:15.068772 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:15.068783 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:15.068929 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 00:07:15.068968 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 00:07:15.069075 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:15.066184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34395 (local address 127.0.98.3:53623)
0620 00:07:15.066344 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:15.066353 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:15.066434 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:15.066596 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:15.066597 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:15.066599 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:07:15.066681 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:15.066683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:15.067325 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:15.067326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:15.068125 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:15.068127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:15.068783 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:15.068784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:15.068862 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:15.068867 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:07:15.068922 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 00:07:15.068965 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":45,"thread_start_us":91,"threads_started":1}
I20260620 00:07:15.069074 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:15.066126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35827 (local address 127.0.98.3:48199)
0620 00:07:15.066279 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:15.066292 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:15.066374 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:15.066548 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:15.066549 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:15.066552 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:07:15.066633 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:15.066635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:15.067274 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:15.067275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:15.068110 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:15.068112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:15.068756 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:15.068758 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:15.068837 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:15.068841 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:07:15.068912 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 00:07:15.068958 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":14,"thread_start_us":86,"threads_started":1}
I20260620 00:07:15.069518 8049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f29ff8aba72e4e95a2d899d2dcb7aee6" candidate_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f34094d8e684686b4004cf910f8f16a" is_pre_election: true
I20260620 00:07:15.069593 8189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f29ff8aba72e4e95a2d899d2dcb7aee6" candidate_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ca88ea258443a39dad68029d288f95" is_pre_election: true
I20260620 00:07:15.069643 8049 raft_consensus.cc:2454] T f29ff8aba72e4e95a2d899d2dcb7aee6 P 6f34094d8e684686b4004cf910f8f16a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9fc1f19fe2a4e4f93eddca1b80f3fb4 in term 0.
I20260620 00:07:15.069698 8189 raft_consensus.cc:2454] T f29ff8aba72e4e95a2d899d2dcb7aee6 P 69ca88ea258443a39dad68029d288f95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9fc1f19fe2a4e4f93eddca1b80f3fb4 in term 0.
I20260620 00:07:15.069821 8261 leader_election.cc:302] T f29ff8aba72e4e95a2d899d2dcb7aee6 P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f34094d8e684686b4004cf910f8f16a, d9fc1f19fe2a4e4f93eddca1b80f3fb4; no voters:
I20260620 00:07:15.070013 8410 raft_consensus.cc:2781] T f29ff8aba72e4e95a2d899d2dcb7aee6 P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:07:15.070068 8410 raft_consensus.cc:482] T f29ff8aba72e4e95a2d899d2dcb7aee6 P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:07:15.070091 8410 raft_consensus.cc:3037] T f29ff8aba72e4e95a2d899d2dcb7aee6 P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:15.071013 8410 raft_consensus.cc:504] T f29ff8aba72e4e95a2d899d2dcb7aee6 P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } }
I20260620 00:07:15.071167 8410 leader_election.cc:288] T f29ff8aba72e4e95a2d899d2dcb7aee6 P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [CANDIDATE]: Term 1 election: Requested vote from peers 6f34094d8e684686b4004cf910f8f16a (127.0.98.1:35827), 69ca88ea258443a39dad68029d288f95 (127.0.98.2:34395)
I20260620 00:07:15.071357 8189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f29ff8aba72e4e95a2d899d2dcb7aee6" candidate_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ca88ea258443a39dad68029d288f95"
I20260620 00:07:15.071461 8189 raft_consensus.cc:3037] T f29ff8aba72e4e95a2d899d2dcb7aee6 P 69ca88ea258443a39dad68029d288f95 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:15.071666 8049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f29ff8aba72e4e95a2d899d2dcb7aee6" candidate_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f34094d8e684686b4004cf910f8f16a"
I20260620 00:07:15.071741 8049 raft_consensus.cc:3037] T f29ff8aba72e4e95a2d899d2dcb7aee6 P 6f34094d8e684686b4004cf910f8f16a [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:15.072274 8189 raft_consensus.cc:2454] T f29ff8aba72e4e95a2d899d2dcb7aee6 P 69ca88ea258443a39dad68029d288f95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9fc1f19fe2a4e4f93eddca1b80f3fb4 in term 1.
I20260620 00:07:15.072479 8263 leader_election.cc:302] T f29ff8aba72e4e95a2d899d2dcb7aee6 P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ca88ea258443a39dad68029d288f95, d9fc1f19fe2a4e4f93eddca1b80f3fb4; no voters:
I20260620 00:07:15.072508 8049 raft_consensus.cc:2454] T f29ff8aba72e4e95a2d899d2dcb7aee6 P 6f34094d8e684686b4004cf910f8f16a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9fc1f19fe2a4e4f93eddca1b80f3fb4 in term 1.
I20260620 00:07:15.072582 8410 raft_consensus.cc:2781] T f29ff8aba72e4e95a2d899d2dcb7aee6 P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:07:15.072703 8410 raft_consensus.cc:686] T f29ff8aba72e4e95a2d899d2dcb7aee6 P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 1 LEADER]: Becoming Leader. State: Replica: d9fc1f19fe2a4e4f93eddca1b80f3fb4, State: Running, Role: LEADER
I20260620 00:07:15.072777 8410 consensus_queue.cc:237] T f29ff8aba72e4e95a2d899d2dcb7aee6 P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } }
I20260620 00:07:15.073083 8447 tablet_replica.cc:439] T f29ff8aba72e4e95a2d899d2dcb7aee6 P d9fc1f19fe2a4e4f93eddca1b80f3fb4: TxnStatusTablet state changed. Reason: New leader d9fc1f19fe2a4e4f93eddca1b80f3fb4. Latest consensus state: current_term: 1 leader_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } } }
I20260620 00:07:15.073184 8447 tablet_replica.cc:442] T f29ff8aba72e4e95a2d899d2dcb7aee6 P d9fc1f19fe2a4e4f93eddca1b80f3fb4: This TxnStatusTablet replica's current role is: LEADER
I20260620 00:07:15.073472 8449 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 00:07:15.073590 8449 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 00:07:15.073554 7891 catalog_manager.cc:5495] T f29ff8aba72e4e95a2d899d2dcb7aee6 P d9fc1f19fe2a4e4f93eddca1b80f3fb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to d9fc1f19fe2a4e4f93eddca1b80f3fb4 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } health_report { overall_health: HEALTHY } } }
I20260620 00:07:15.130698 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:15.128150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44302 (local address 127.0.98.3:42519)
0620 00:07:15.128217 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:15.128219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:15.128228 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:07:15.128308 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:15.128309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:15.128327 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:15.128389 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:07:15.128546 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:15.129028 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:15.129860 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:15.130071 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:15.130157 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:15.130350 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:07:15.130458 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:07:15.130540 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:15.130596 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 00:07:15.130616 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 00:07:15.130712 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:15.128066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42519 (local address 127.0.0.1:44302)
0620 00:07:15.128150 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:15.128163 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:15.128212 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:15.128411 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:15.128412 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:15.128414 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 00:07:15.128468 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:15.128471 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:15.129116 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:15.129118 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:15.129763 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:15.129765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:15.130160 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:15.130161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:15.130228 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:15.130242 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 00:07:15.130526 (+ 284us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 00:07:15.130535 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 00:07:15.130590 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 00:07:15.130644 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 00:07:15.131987 8189 raft_consensus.cc:1261] T f29ff8aba72e4e95a2d899d2dcb7aee6 P 69ca88ea258443a39dad68029d288f95 [term 1 FOLLOWER]: Refusing update from remote peer d9fc1f19fe2a4e4f93eddca1b80f3fb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:07:15.132082 8049 raft_consensus.cc:1261] T f29ff8aba72e4e95a2d899d2dcb7aee6 P 6f34094d8e684686b4004cf910f8f16a [term 1 FOLLOWER]: Refusing update from remote peer d9fc1f19fe2a4e4f93eddca1b80f3fb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:07:15.132285 8447 consensus_queue.cc:1035] T f29ff8aba72e4e95a2d899d2dcb7aee6 P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:07:15.132400 8410 consensus_queue.cc:1035] T f29ff8aba72e4e95a2d899d2dcb7aee6 P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:07:15.132967 8409 tablet_replica.cc:439] T f29ff8aba72e4e95a2d899d2dcb7aee6 P 69ca88ea258443a39dad68029d288f95: TxnStatusTablet state changed. Reason: New leader d9fc1f19fe2a4e4f93eddca1b80f3fb4. Latest consensus state: current_term: 1 leader_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } } }
I20260620 00:07:15.133069 8409 tablet_replica.cc:442] T f29ff8aba72e4e95a2d899d2dcb7aee6 P 69ca88ea258443a39dad68029d288f95: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:07:15.133163 8423 tablet_replica.cc:439] T f29ff8aba72e4e95a2d899d2dcb7aee6 P 6f34094d8e684686b4004cf910f8f16a: TxnStatusTablet state changed. Reason: New leader d9fc1f19fe2a4e4f93eddca1b80f3fb4. Latest consensus state: current_term: 1 leader_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } } }
I20260620 00:07:15.133268 8423 tablet_replica.cc:442] T f29ff8aba72e4e95a2d899d2dcb7aee6 P 6f34094d8e684686b4004cf910f8f16a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:07:15.133987 8410 tablet_replica.cc:439] T f29ff8aba72e4e95a2d899d2dcb7aee6 P d9fc1f19fe2a4e4f93eddca1b80f3fb4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } } }
I20260620 00:07:15.134085 8410 tablet_replica.cc:442] T f29ff8aba72e4e95a2d899d2dcb7aee6 P d9fc1f19fe2a4e4f93eddca1b80f3fb4: This TxnStatusTablet replica's current role is: LEADER
I20260620 00:07:15.134650 8447 tablet_replica.cc:439] T f29ff8aba72e4e95a2d899d2dcb7aee6 P d9fc1f19fe2a4e4f93eddca1b80f3fb4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } } }
I20260620 00:07:15.134671 8423 tablet_replica.cc:439] T f29ff8aba72e4e95a2d899d2dcb7aee6 P 6f34094d8e684686b4004cf910f8f16a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } } }
I20260620 00:07:15.134735 8447 tablet_replica.cc:442] T f29ff8aba72e4e95a2d899d2dcb7aee6 P d9fc1f19fe2a4e4f93eddca1b80f3fb4: This TxnStatusTablet replica's current role is: LEADER
I20260620 00:07:15.134750 8423 tablet_replica.cc:442] T f29ff8aba72e4e95a2d899d2dcb7aee6 P 6f34094d8e684686b4004cf910f8f16a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:07:15.134986 8409 tablet_replica.cc:439] T f29ff8aba72e4e95a2d899d2dcb7aee6 P 69ca88ea258443a39dad68029d288f95: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } } }
I20260620 00:07:15.135079 8409 tablet_replica.cc:442] T f29ff8aba72e4e95a2d899d2dcb7aee6 P 69ca88ea258443a39dad68029d288f95: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:07:15.138398 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:15.136011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47520 (local address 127.0.98.2:34395)
0620 00:07:15.136052 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:15.136054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:15.136061 (+ 7us) server_negotiation.cc:407] Connection header received
0620 00:07:15.136096 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:15.136097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:15.136115 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:15.136203 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:07:15.136316 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:15.136786 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:15.137616 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:15.137824 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:15.137903 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:15.138049 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:07:15.138156 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:07:15.138237 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:15.138296 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 00:07:15.138316 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 00:07:15.144083 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:15.141538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55699 (local address 127.0.98.3:42519)
0620 00:07:15.141630 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:15.141632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:15.141700 (+ 68us) server_negotiation.cc:407] Connection header received
0620 00:07:15.141788 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:15.141790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:15.141801 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:15.141859 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:07:15.141987 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:15.142531 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:15.143326 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:15.143525 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:15.143604 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:15.143807 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:07:15.143846 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:07:15.143911 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:15.143980 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 00:07:15.143999 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 00:07:15.144110 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:15.141339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42519 (local address 127.0.98.2:55699)
0620 00:07:15.141635 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:15.141647 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:15.141719 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:15.141860 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:15.141861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:15.141864 (+ 3us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 00:07:15.141924 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:15.141925 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:15.142601 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:15.142602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:15.143228 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:15.143231 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:15.143632 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:15.143633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:15.143696 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:15.143710 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 00:07:15.143912 (+ 202us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 00:07:15.143914 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 00:07:15.143976 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 00:07:15.144019 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":140,"threads_started":1}
I20260620 00:07:15.147010 8329 raft_consensus.cc:1261] T 2334f24224a84dd3849cff0f57142b0e P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 1 FOLLOWER]: Refusing update from remote peer 69ca88ea258443a39dad68029d288f95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:07:15.147056 8049 raft_consensus.cc:1261] T 2334f24224a84dd3849cff0f57142b0e P 6f34094d8e684686b4004cf910f8f16a [term 1 FOLLOWER]: Refusing update from remote peer 69ca88ea258443a39dad68029d288f95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:07:15.147294 8434 consensus_queue.cc:1035] T 2334f24224a84dd3849cff0f57142b0e P 69ca88ea258443a39dad68029d288f95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:07:15.147420 8409 consensus_queue.cc:1035] T 2334f24224a84dd3849cff0f57142b0e P 69ca88ea258443a39dad68029d288f95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:07:15.156597 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:15.153744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51537 (local address 127.0.98.2:34395)
0620 00:07:15.153800 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:15.153802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:15.153888 (+ 86us) server_negotiation.cc:407] Connection header received
0620 00:07:15.153966 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:15.153967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:15.153977 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:15.154026 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:07:15.154223 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:15.154694 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:15.155802 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:15.156047 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:15.156102 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:15.156341 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:07:15.156377 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:07:15.156408 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:15.156508 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 00:07:15.156536 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 00:07:15.156646 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:15.153672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34395 (local address 127.0.98.3:51537)
0620 00:07:15.153824 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:15.153837 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:15.153898 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:15.154075 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:15.154076 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:15.154078 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 00:07:15.154151 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:15.154153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:15.154802 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:15.154803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:15.155707 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:15.155710 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:15.156138 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:15.156140 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:15.156210 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:15.156228 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 00:07:15.156451 (+ 223us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 00:07:15.156452 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 00:07:15.156491 (+ 39us) client_negotiation.cc:240] Negotiation successful
0620 00:07:15.156538 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
I20260620 00:07:15.167707 7891 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:07:15.167847 7891 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:07:15.168486 7891 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05cccc5116e84b959eee49c8c6ca05a4: Sending DeleteTablet for 3 replicas of tablet 2334f24224a84dd3849cff0f57142b0e
I20260620 00:07:15.168838 8029 tablet_service.cc:1514] Processing DeleteTablet for tablet 2334f24224a84dd3849cff0f57142b0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:07:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35558
I20260620 00:07:15.168877 8169 tablet_service.cc:1514] Processing DeleteTablet for tablet 2334f24224a84dd3849cff0f57142b0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:07:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47518
I20260620 00:07:15.168990 8427 tablet_replica.cc:330] T 2334f24224a84dd3849cff0f57142b0e P 6f34094d8e684686b4004cf910f8f16a: stopping tablet replica
I20260620 00:07:15.169054 8427 raft_consensus.cc:2229] T 2334f24224a84dd3849cff0f57142b0e P 6f34094d8e684686b4004cf910f8f16a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:07:15.169121 8427 raft_consensus.cc:2258] T 2334f24224a84dd3849cff0f57142b0e P 6f34094d8e684686b4004cf910f8f16a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:07:15.169196 8428 tablet_replica.cc:330] T 2334f24224a84dd3849cff0f57142b0e P 69ca88ea258443a39dad68029d288f95: stopping tablet replica
I20260620 00:07:15.169247 8428 raft_consensus.cc:2229] T 2334f24224a84dd3849cff0f57142b0e P 69ca88ea258443a39dad68029d288f95 [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:07:15.169402 8428 raft_consensus.cc:2258] T 2334f24224a84dd3849cff0f57142b0e P 69ca88ea258443a39dad68029d288f95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:07:15.169553 8309 tablet_service.cc:1514] Processing DeleteTablet for tablet 2334f24224a84dd3849cff0f57142b0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:07:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44294
I20260620 00:07:15.169644 8429 tablet_replica.cc:330] T 2334f24224a84dd3849cff0f57142b0e P d9fc1f19fe2a4e4f93eddca1b80f3fb4: stopping tablet replica
I20260620 00:07:15.169694 8429 raft_consensus.cc:2229] T 2334f24224a84dd3849cff0f57142b0e P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:07:15.169735 8429 raft_consensus.cc:2258] T 2334f24224a84dd3849cff0f57142b0e P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:07:15.170100 8429 ts_tablet_manager.cc:1905] T 2334f24224a84dd3849cff0f57142b0e P d9fc1f19fe2a4e4f93eddca1b80f3fb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:07:15.170217 8428 ts_tablet_manager.cc:1905] T 2334f24224a84dd3849cff0f57142b0e P 69ca88ea258443a39dad68029d288f95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:07:15.171994 8429 ts_tablet_manager.cc:1918] T 2334f24224a84dd3849cff0f57142b0e P d9fc1f19fe2a4e4f93eddca1b80f3fb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 00:07:15.171994 8428 ts_tablet_manager.cc:1918] T 2334f24224a84dd3849cff0f57142b0e P 69ca88ea258443a39dad68029d288f95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 00:07:15.172053 8429 log.cc:1196] T 2334f24224a84dd3849cff0f57142b0e P d9fc1f19fe2a4e4f93eddca1b80f3fb4: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-2/wal/wals/2334f24224a84dd3849cff0f57142b0e
I20260620 00:07:15.172053 8428 log.cc:1196] T 2334f24224a84dd3849cff0f57142b0e P 69ca88ea258443a39dad68029d288f95: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-1/wal/wals/2334f24224a84dd3849cff0f57142b0e
I20260620 00:07:15.172276 8429 ts_tablet_manager.cc:1939] T 2334f24224a84dd3849cff0f57142b0e P d9fc1f19fe2a4e4f93eddca1b80f3fb4: Deleting consensus metadata
I20260620 00:07:15.172305 8428 ts_tablet_manager.cc:1939] T 2334f24224a84dd3849cff0f57142b0e P 69ca88ea258443a39dad68029d288f95: Deleting consensus metadata
I20260620 00:07:15.172593 7878 catalog_manager.cc:4841] TS d9fc1f19fe2a4e4f93eddca1b80f3fb4 (127.0.98.3:42519): tablet 2334f24224a84dd3849cff0f57142b0e (table test-table [id=14439cf54eef4227bd5adaa2b224afba]) successfully deleted
I20260620 00:07:15.174063 7879 catalog_manager.cc:4841] TS 69ca88ea258443a39dad68029d288f95 (127.0.98.2:34395): tablet 2334f24224a84dd3849cff0f57142b0e (table test-table [id=14439cf54eef4227bd5adaa2b224afba]) successfully deleted
I20260620 00:07:15.175236 8427 ts_tablet_manager.cc:1905] T 2334f24224a84dd3849cff0f57142b0e P 6f34094d8e684686b4004cf910f8f16a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:07:15.177134 8427 ts_tablet_manager.cc:1918] T 2334f24224a84dd3849cff0f57142b0e P 6f34094d8e684686b4004cf910f8f16a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 00:07:15.177208 8427 log.cc:1196] T 2334f24224a84dd3849cff0f57142b0e P 6f34094d8e684686b4004cf910f8f16a: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-0/wal/wals/2334f24224a84dd3849cff0f57142b0e
I20260620 00:07:15.177486 8427 ts_tablet_manager.cc:1939] T 2334f24224a84dd3849cff0f57142b0e P 6f34094d8e684686b4004cf910f8f16a: Deleting consensus metadata
I20260620 00:07:15.177868 7877 catalog_manager.cc:4841] TS 6f34094d8e684686b4004cf910f8f16a (127.0.98.1:35827): tablet 2334f24224a84dd3849cff0f57142b0e (table test-table [id=14439cf54eef4227bd5adaa2b224afba]) successfully deleted
Jun 20 00:07:15 dist-test-slave-grqt krb5kdc[7847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 00:07:15 dist-test-slave-grqt krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:15 dist-test-slave-grqt krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 00:07:15.204828 7959 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:15.197243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35566 (local address 127.0.98.62:38645)
0620 00:07:15.197292 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:15.197294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:15.197686 (+ 392us) server_negotiation.cc:407] Connection header received
0620 00:07:15.197804 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:15.197806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:15.197818 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:15.197850 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:15.198653 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:15.199385 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:15.200524 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:15.200836 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:15.200883 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:15.204756 (+ 3873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35566: BlockingRecv error: recv got EOF from 127.0.0.1:35566 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Jun 20 00:07:15 dist-test-slave-grqt krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 00:07:15.215618 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:15.206251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35570 (local address 127.0.98.62:38645)
0620 00:07:15.206295 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:15.206297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:15.206434 (+ 137us) server_negotiation.cc:407] Connection header received
0620 00:07:15.206493 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:15.206494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:15.206506 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:15.206568 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:15.207174 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:15.207962 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:15.208993 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:15.209318 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:15.209421 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:15.212356 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:15.212358 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:15.212358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:15.212367 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:15.214375 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:15.214794 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:15.214796 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:15.214796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:15.214859 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:15.215074 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:15.215075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:15.215076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:15.215218 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:15.215323 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:15.215437 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 00:07:15.215517 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 00:07:15.217445 7891 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:07:15.217631 7891 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:07:15.218901 8169 tablet_service.cc:1467] Processing CreateTablet for tablet 513716acdbf445608f9c8572a79d22a3 (DEFAULT_TABLE table=test-table [id=5b5af2b58e51469c93391b502f585417]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:07:15.218935 8029 tablet_service.cc:1467] Processing CreateTablet for tablet 513716acdbf445608f9c8572a79d22a3 (DEFAULT_TABLE table=test-table [id=5b5af2b58e51469c93391b502f585417]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:07:15.219036 8169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 513716acdbf445608f9c8572a79d22a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:15.219061 8029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 513716acdbf445608f9c8572a79d22a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:15.220286 8403 tablet_bootstrap.cc:492] T 513716acdbf445608f9c8572a79d22a3 P 69ca88ea258443a39dad68029d288f95: Bootstrap starting.
I20260620 00:07:15.220414 8309 tablet_service.cc:1467] Processing CreateTablet for tablet 513716acdbf445608f9c8572a79d22a3 (DEFAULT_TABLE table=test-table [id=5b5af2b58e51469c93391b502f585417]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:07:15.220530 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 513716acdbf445608f9c8572a79d22a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:15.220681 8403 tablet_bootstrap.cc:654] T 513716acdbf445608f9c8572a79d22a3 P 69ca88ea258443a39dad68029d288f95: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:15.221437 8405 tablet_bootstrap.cc:492] T 513716acdbf445608f9c8572a79d22a3 P 6f34094d8e684686b4004cf910f8f16a: Bootstrap starting.
I20260620 00:07:15.221593 8403 tablet_bootstrap.cc:492] T 513716acdbf445608f9c8572a79d22a3 P 69ca88ea258443a39dad68029d288f95: No bootstrap required, opened a new log
I20260620 00:07:15.221633 8403 ts_tablet_manager.cc:1397] T 513716acdbf445608f9c8572a79d22a3 P 69ca88ea258443a39dad68029d288f95: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 00:07:15.221756 8405 tablet_bootstrap.cc:654] T 513716acdbf445608f9c8572a79d22a3 P 6f34094d8e684686b4004cf910f8f16a: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:15.221772 8403 raft_consensus.cc:348] T 513716acdbf445608f9c8572a79d22a3 P 69ca88ea258443a39dad68029d288f95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } }
I20260620 00:07:15.221832 8403 raft_consensus.cc:374] T 513716acdbf445608f9c8572a79d22a3 P 69ca88ea258443a39dad68029d288f95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:15.221860 8403 raft_consensus.cc:729] T 513716acdbf445608f9c8572a79d22a3 P 69ca88ea258443a39dad68029d288f95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ca88ea258443a39dad68029d288f95, State: Initialized, Role: FOLLOWER
I20260620 00:07:15.221900 8403 consensus_queue.cc:260] T 513716acdbf445608f9c8572a79d22a3 P 69ca88ea258443a39dad68029d288f95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } }
I20260620 00:07:15.221987 8403 ts_tablet_manager.cc:1428] T 513716acdbf445608f9c8572a79d22a3 P 69ca88ea258443a39dad68029d288f95: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:15.222736 8404 tablet_bootstrap.cc:492] T 513716acdbf445608f9c8572a79d22a3 P d9fc1f19fe2a4e4f93eddca1b80f3fb4: Bootstrap starting.
I20260620 00:07:15.223152 8404 tablet_bootstrap.cc:654] T 513716acdbf445608f9c8572a79d22a3 P d9fc1f19fe2a4e4f93eddca1b80f3fb4: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:15.225095 8405 tablet_bootstrap.cc:492] T 513716acdbf445608f9c8572a79d22a3 P 6f34094d8e684686b4004cf910f8f16a: No bootstrap required, opened a new log
I20260620 00:07:15.225106 8404 tablet_bootstrap.cc:492] T 513716acdbf445608f9c8572a79d22a3 P d9fc1f19fe2a4e4f93eddca1b80f3fb4: No bootstrap required, opened a new log
I20260620 00:07:15.225149 8404 ts_tablet_manager.cc:1397] T 513716acdbf445608f9c8572a79d22a3 P d9fc1f19fe2a4e4f93eddca1b80f3fb4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:07:15.225148 8405 ts_tablet_manager.cc:1397] T 513716acdbf445608f9c8572a79d22a3 P 6f34094d8e684686b4004cf910f8f16a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 00:07:15.225256 8405 raft_consensus.cc:348] T 513716acdbf445608f9c8572a79d22a3 P 6f34094d8e684686b4004cf910f8f16a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } }
I20260620 00:07:15.225298 8404 raft_consensus.cc:348] T 513716acdbf445608f9c8572a79d22a3 P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } }
I20260620 00:07:15.225426 8404 raft_consensus.cc:374] T 513716acdbf445608f9c8572a79d22a3 P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:15.225427 8405 raft_consensus.cc:374] T 513716acdbf445608f9c8572a79d22a3 P 6f34094d8e684686b4004cf910f8f16a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:15.225448 8404 raft_consensus.cc:729] T 513716acdbf445608f9c8572a79d22a3 P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9fc1f19fe2a4e4f93eddca1b80f3fb4, State: Initialized, Role: FOLLOWER
I20260620 00:07:15.225456 8405 raft_consensus.cc:729] T 513716acdbf445608f9c8572a79d22a3 P 6f34094d8e684686b4004cf910f8f16a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f34094d8e684686b4004cf910f8f16a, State: Initialized, Role: FOLLOWER
I20260620 00:07:15.225490 8404 consensus_queue.cc:260] T 513716acdbf445608f9c8572a79d22a3 P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } }
I20260620 00:07:15.225500 8405 consensus_queue.cc:260] T 513716acdbf445608f9c8572a79d22a3 P 6f34094d8e684686b4004cf910f8f16a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } }
I20260620 00:07:15.225572 8404 ts_tablet_manager.cc:1428] T 513716acdbf445608f9c8572a79d22a3 P d9fc1f19fe2a4e4f93eddca1b80f3fb4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:15.225601 8405 ts_tablet_manager.cc:1428] T 513716acdbf445608f9c8572a79d22a3 P 6f34094d8e684686b4004cf910f8f16a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 00:07:15.589790 8423 raft_consensus.cc:482] T 513716acdbf445608f9c8572a79d22a3 P 6f34094d8e684686b4004cf910f8f16a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:07:15.589926 8423 raft_consensus.cc:504] T 513716acdbf445608f9c8572a79d22a3 P 6f34094d8e684686b4004cf910f8f16a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } }
I20260620 00:07:15.590137 8423 leader_election.cc:288] T 513716acdbf445608f9c8572a79d22a3 P 6f34094d8e684686b4004cf910f8f16a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69ca88ea258443a39dad68029d288f95 (127.0.98.2:34395), d9fc1f19fe2a4e4f93eddca1b80f3fb4 (127.0.98.3:42519)
I20260620 00:07:15.590399 8189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "513716acdbf445608f9c8572a79d22a3" candidate_uuid: "6f34094d8e684686b4004cf910f8f16a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ca88ea258443a39dad68029d288f95" is_pre_election: true
I20260620 00:07:15.590433 8329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "513716acdbf445608f9c8572a79d22a3" candidate_uuid: "6f34094d8e684686b4004cf910f8f16a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" is_pre_election: true
I20260620 00:07:15.590520 8189 raft_consensus.cc:2454] T 513716acdbf445608f9c8572a79d22a3 P 69ca88ea258443a39dad68029d288f95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f34094d8e684686b4004cf910f8f16a in term 0.
I20260620 00:07:15.590515 8329 raft_consensus.cc:2454] T 513716acdbf445608f9c8572a79d22a3 P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f34094d8e684686b4004cf910f8f16a in term 0.
I20260620 00:07:15.590684 7982 leader_election.cc:302] T 513716acdbf445608f9c8572a79d22a3 P 6f34094d8e684686b4004cf910f8f16a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f34094d8e684686b4004cf910f8f16a, d9fc1f19fe2a4e4f93eddca1b80f3fb4; no voters:
I20260620 00:07:15.590804 8423 raft_consensus.cc:2781] T 513716acdbf445608f9c8572a79d22a3 P 6f34094d8e684686b4004cf910f8f16a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:07:15.590857 8423 raft_consensus.cc:482] T 513716acdbf445608f9c8572a79d22a3 P 6f34094d8e684686b4004cf910f8f16a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:07:15.590878 8423 raft_consensus.cc:3037] T 513716acdbf445608f9c8572a79d22a3 P 6f34094d8e684686b4004cf910f8f16a [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:15.591609 8423 raft_consensus.cc:504] T 513716acdbf445608f9c8572a79d22a3 P 6f34094d8e684686b4004cf910f8f16a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } }
I20260620 00:07:15.591784 8423 leader_election.cc:288] T 513716acdbf445608f9c8572a79d22a3 P 6f34094d8e684686b4004cf910f8f16a [CANDIDATE]: Term 1 election: Requested vote from peers 69ca88ea258443a39dad68029d288f95 (127.0.98.2:34395), d9fc1f19fe2a4e4f93eddca1b80f3fb4 (127.0.98.3:42519)
I20260620 00:07:15.591955 8189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "513716acdbf445608f9c8572a79d22a3" candidate_uuid: "6f34094d8e684686b4004cf910f8f16a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ca88ea258443a39dad68029d288f95"
I20260620 00:07:15.591946 8329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "513716acdbf445608f9c8572a79d22a3" candidate_uuid: "6f34094d8e684686b4004cf910f8f16a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4"
I20260620 00:07:15.592026 8189 raft_consensus.cc:3037] T 513716acdbf445608f9c8572a79d22a3 P 69ca88ea258443a39dad68029d288f95 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:15.592048 8329 raft_consensus.cc:3037] T 513716acdbf445608f9c8572a79d22a3 P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:15.592661 8189 raft_consensus.cc:2454] T 513716acdbf445608f9c8572a79d22a3 P 69ca88ea258443a39dad68029d288f95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f34094d8e684686b4004cf910f8f16a in term 1.
I20260620 00:07:15.592785 8329 raft_consensus.cc:2454] T 513716acdbf445608f9c8572a79d22a3 P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f34094d8e684686b4004cf910f8f16a in term 1.
I20260620 00:07:15.592873 7983 leader_election.cc:302] T 513716acdbf445608f9c8572a79d22a3 P 6f34094d8e684686b4004cf910f8f16a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ca88ea258443a39dad68029d288f95, 6f34094d8e684686b4004cf910f8f16a; no voters:
I20260620 00:07:15.592970 8423 raft_consensus.cc:2781] T 513716acdbf445608f9c8572a79d22a3 P 6f34094d8e684686b4004cf910f8f16a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:07:15.593096 8423 raft_consensus.cc:686] T 513716acdbf445608f9c8572a79d22a3 P 6f34094d8e684686b4004cf910f8f16a [term 1 LEADER]: Becoming Leader. State: Replica: 6f34094d8e684686b4004cf910f8f16a, State: Running, Role: LEADER
I20260620 00:07:15.593152 8423 consensus_queue.cc:237] T 513716acdbf445608f9c8572a79d22a3 P 6f34094d8e684686b4004cf910f8f16a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } }
I20260620 00:07:15.593920 7892 catalog_manager.cc:5495] T 513716acdbf445608f9c8572a79d22a3 P 6f34094d8e684686b4004cf910f8f16a reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f34094d8e684686b4004cf910f8f16a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6f34094d8e684686b4004cf910f8f16a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } health_report { overall_health: UNKNOWN } } }
I20260620 00:07:15.668754 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:15.666482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35586 (local address 127.0.98.1:35827)
0620 00:07:15.666598 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:15.666600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:15.666608 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:07:15.666630 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:15.666631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:15.666642 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:15.666706 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:07:15.666821 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:15.667304 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:15.668072 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:15.668310 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:15.668368 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:15.668498 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:07:15.668525 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:07:15.668564 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:15.668612 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 00:07:15.668630 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":69,"threads_started":1}
I20260620 00:07:15.669834 8329 raft_consensus.cc:1261] T 513716acdbf445608f9c8572a79d22a3 P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 1 FOLLOWER]: Refusing update from remote peer 6f34094d8e684686b4004cf910f8f16a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:07:15.669932 8189 raft_consensus.cc:1261] T 513716acdbf445608f9c8572a79d22a3 P 69ca88ea258443a39dad68029d288f95 [term 1 FOLLOWER]: Refusing update from remote peer 6f34094d8e684686b4004cf910f8f16a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:07:15.670117 8423 consensus_queue.cc:1035] T 513716acdbf445608f9c8572a79d22a3 P 6f34094d8e684686b4004cf910f8f16a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:07:15.670329 8480 consensus_queue.cc:1035] T 513716acdbf445608f9c8572a79d22a3 P 6f34094d8e684686b4004cf910f8f16a [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:07:15.672724 7892 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:07:15.672820 7892 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:07:15.673504 7892 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05cccc5116e84b959eee49c8c6ca05a4: Sending DeleteTablet for 3 replicas of tablet 513716acdbf445608f9c8572a79d22a3
I20260620 00:07:15.673771 8169 tablet_service.cc:1514] Processing DeleteTablet for tablet 513716acdbf445608f9c8572a79d22a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:07:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47518
I20260620 00:07:15.673784 8309 tablet_service.cc:1514] Processing DeleteTablet for tablet 513716acdbf445608f9c8572a79d22a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:07:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44294
I20260620 00:07:15.673820 8029 tablet_service.cc:1514] Processing DeleteTablet for tablet 513716acdbf445608f9c8572a79d22a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:07:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35558
I20260620 00:07:15.673931 8492 tablet_replica.cc:330] T 513716acdbf445608f9c8572a79d22a3 P 69ca88ea258443a39dad68029d288f95: stopping tablet replica
I20260620 00:07:15.673931 8427 tablet_replica.cc:330] T 513716acdbf445608f9c8572a79d22a3 P 6f34094d8e684686b4004cf910f8f16a: stopping tablet replica
I20260620 00:07:15.673986 8493 tablet_replica.cc:330] T 513716acdbf445608f9c8572a79d22a3 P d9fc1f19fe2a4e4f93eddca1b80f3fb4: stopping tablet replica
I20260620 00:07:15.673996 8427 raft_consensus.cc:2229] T 513716acdbf445608f9c8572a79d22a3 P 6f34094d8e684686b4004cf910f8f16a [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:07:15.673995 8492 raft_consensus.cc:2229] T 513716acdbf445608f9c8572a79d22a3 P 69ca88ea258443a39dad68029d288f95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:07:15.674043 8492 raft_consensus.cc:2258] T 513716acdbf445608f9c8572a79d22a3 P 69ca88ea258443a39dad68029d288f95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:07:15.674053 8427 raft_consensus.cc:2258] T 513716acdbf445608f9c8572a79d22a3 P 6f34094d8e684686b4004cf910f8f16a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:07:15.674166 8493 raft_consensus.cc:2229] T 513716acdbf445608f9c8572a79d22a3 P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:07:15.674253 8493 raft_consensus.cc:2258] T 513716acdbf445608f9c8572a79d22a3 P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:07:15.674348 8427 ts_tablet_manager.cc:1905] T 513716acdbf445608f9c8572a79d22a3 P 6f34094d8e684686b4004cf910f8f16a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:07:15.674629 8493 ts_tablet_manager.cc:1905] T 513716acdbf445608f9c8572a79d22a3 P d9fc1f19fe2a4e4f93eddca1b80f3fb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:07:15.674595 7892 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:07:15.674763 7892 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:07:15.676115 8427 ts_tablet_manager.cc:1918] T 513716acdbf445608f9c8572a79d22a3 P 6f34094d8e684686b4004cf910f8f16a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:07:15.676138 8492 ts_tablet_manager.cc:1905] T 513716acdbf445608f9c8572a79d22a3 P 69ca88ea258443a39dad68029d288f95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:07:15.676184 8029 tablet_service.cc:1467] Processing CreateTablet for tablet 9e71fe92ef174285966b215575c333fd (DEFAULT_TABLE table=test-table [id=6103af9cab9e4e98bee063136a830010]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:07:15.676213 8169 tablet_service.cc:1467] Processing CreateTablet for tablet 9e71fe92ef174285966b215575c333fd (DEFAULT_TABLE table=test-table [id=6103af9cab9e4e98bee063136a830010]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:07:15.676306 8029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e71fe92ef174285966b215575c333fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:15.676354 8169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e71fe92ef174285966b215575c333fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:15.676676 8493 ts_tablet_manager.cc:1918] T 513716acdbf445608f9c8572a79d22a3 P d9fc1f19fe2a4e4f93eddca1b80f3fb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:07:15.676726 8493 log.cc:1196] T 513716acdbf445608f9c8572a79d22a3 P d9fc1f19fe2a4e4f93eddca1b80f3fb4: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-2/wal/wals/513716acdbf445608f9c8572a79d22a3
I20260620 00:07:15.676960 8493 ts_tablet_manager.cc:1939] T 513716acdbf445608f9c8572a79d22a3 P d9fc1f19fe2a4e4f93eddca1b80f3fb4: Deleting consensus metadata
I20260620 00:07:15.677565 8405 tablet_bootstrap.cc:492] T 9e71fe92ef174285966b215575c333fd P 6f34094d8e684686b4004cf910f8f16a: Bootstrap starting.
I20260620 00:07:15.677644 8403 tablet_bootstrap.cc:492] T 9e71fe92ef174285966b215575c333fd P 69ca88ea258443a39dad68029d288f95: Bootstrap starting.
I20260620 00:07:15.677686 7878 catalog_manager.cc:4841] TS d9fc1f19fe2a4e4f93eddca1b80f3fb4 (127.0.98.3:42519): tablet 513716acdbf445608f9c8572a79d22a3 (table test-table [id=5b5af2b58e51469c93391b502f585417]) successfully deleted
I20260620 00:07:15.677855 8309 tablet_service.cc:1467] Processing CreateTablet for tablet 9e71fe92ef174285966b215575c333fd (DEFAULT_TABLE table=test-table [id=6103af9cab9e4e98bee063136a830010]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:07:15.677865 8492 ts_tablet_manager.cc:1918] T 513716acdbf445608f9c8572a79d22a3 P 69ca88ea258443a39dad68029d288f95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:07:15.677914 8492 log.cc:1196] T 513716acdbf445608f9c8572a79d22a3 P 69ca88ea258443a39dad68029d288f95: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-1/wal/wals/513716acdbf445608f9c8572a79d22a3
I20260620 00:07:15.677950 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e71fe92ef174285966b215575c333fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:15.677968 8403 tablet_bootstrap.cc:654] T 9e71fe92ef174285966b215575c333fd P 69ca88ea258443a39dad68029d288f95: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:15.678042 8405 tablet_bootstrap.cc:654] T 9e71fe92ef174285966b215575c333fd P 6f34094d8e684686b4004cf910f8f16a: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:15.678151 8492 ts_tablet_manager.cc:1939] T 513716acdbf445608f9c8572a79d22a3 P 69ca88ea258443a39dad68029d288f95: Deleting consensus metadata
I20260620 00:07:15.676196 8427 log.cc:1196] T 513716acdbf445608f9c8572a79d22a3 P 6f34094d8e684686b4004cf910f8f16a: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-0/wal/wals/513716acdbf445608f9c8572a79d22a3
I20260620 00:07:15.678539 8427 ts_tablet_manager.cc:1939] T 513716acdbf445608f9c8572a79d22a3 P 6f34094d8e684686b4004cf910f8f16a: Deleting consensus metadata
I20260620 00:07:15.678851 7877 catalog_manager.cc:4841] TS 6f34094d8e684686b4004cf910f8f16a (127.0.98.1:35827): tablet 513716acdbf445608f9c8572a79d22a3 (table test-table [id=5b5af2b58e51469c93391b502f585417]) successfully deleted
I20260620 00:07:15.679141 8405 tablet_bootstrap.cc:492] T 9e71fe92ef174285966b215575c333fd P 6f34094d8e684686b4004cf910f8f16a: No bootstrap required, opened a new log
I20260620 00:07:15.679186 8405 ts_tablet_manager.cc:1397] T 9e71fe92ef174285966b215575c333fd P 6f34094d8e684686b4004cf910f8f16a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:07:15.679268 8404 tablet_bootstrap.cc:492] T 9e71fe92ef174285966b215575c333fd P d9fc1f19fe2a4e4f93eddca1b80f3fb4: Bootstrap starting.
I20260620 00:07:15.679301 8405 raft_consensus.cc:348] T 9e71fe92ef174285966b215575c333fd P 6f34094d8e684686b4004cf910f8f16a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } }
I20260620 00:07:15.679380 8405 raft_consensus.cc:374] T 9e71fe92ef174285966b215575c333fd P 6f34094d8e684686b4004cf910f8f16a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:15.679401 8405 raft_consensus.cc:729] T 9e71fe92ef174285966b215575c333fd P 6f34094d8e684686b4004cf910f8f16a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f34094d8e684686b4004cf910f8f16a, State: Initialized, Role: FOLLOWER
I20260620 00:07:15.679414 7879 catalog_manager.cc:4841] TS 69ca88ea258443a39dad68029d288f95 (127.0.98.2:34395): tablet 513716acdbf445608f9c8572a79d22a3 (table test-table [id=5b5af2b58e51469c93391b502f585417]) successfully deleted
I20260620 00:07:15.679450 8405 consensus_queue.cc:260] T 9e71fe92ef174285966b215575c333fd P 6f34094d8e684686b4004cf910f8f16a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } }
I20260620 00:07:15.679563 8405 ts_tablet_manager.cc:1428] T 9e71fe92ef174285966b215575c333fd P 6f34094d8e684686b4004cf910f8f16a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:15.679634 8404 tablet_bootstrap.cc:654] T 9e71fe92ef174285966b215575c333fd P d9fc1f19fe2a4e4f93eddca1b80f3fb4: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:15.680121 8403 tablet_bootstrap.cc:492] T 9e71fe92ef174285966b215575c333fd P 69ca88ea258443a39dad68029d288f95: No bootstrap required, opened a new log
I20260620 00:07:15.680189 8403 ts_tablet_manager.cc:1397] T 9e71fe92ef174285966b215575c333fd P 69ca88ea258443a39dad68029d288f95: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 00:07:15.680325 8403 raft_consensus.cc:348] T 9e71fe92ef174285966b215575c333fd P 69ca88ea258443a39dad68029d288f95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } }
I20260620 00:07:15.680428 8403 raft_consensus.cc:374] T 9e71fe92ef174285966b215575c333fd P 69ca88ea258443a39dad68029d288f95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:15.680450 8403 raft_consensus.cc:729] T 9e71fe92ef174285966b215575c333fd P 69ca88ea258443a39dad68029d288f95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ca88ea258443a39dad68029d288f95, State: Initialized, Role: FOLLOWER
I20260620 00:07:15.680495 8403 consensus_queue.cc:260] T 9e71fe92ef174285966b215575c333fd P 69ca88ea258443a39dad68029d288f95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } }
I20260620 00:07:15.680588 8403 ts_tablet_manager.cc:1428] T 9e71fe92ef174285966b215575c333fd P 69ca88ea258443a39dad68029d288f95: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:15.680787 8404 tablet_bootstrap.cc:492] T 9e71fe92ef174285966b215575c333fd P d9fc1f19fe2a4e4f93eddca1b80f3fb4: No bootstrap required, opened a new log
I20260620 00:07:15.680830 8404 ts_tablet_manager.cc:1397] T 9e71fe92ef174285966b215575c333fd P d9fc1f19fe2a4e4f93eddca1b80f3fb4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:07:15.680958 8404 raft_consensus.cc:348] T 9e71fe92ef174285966b215575c333fd P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } }
I20260620 00:07:15.681028 8404 raft_consensus.cc:374] T 9e71fe92ef174285966b215575c333fd P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:15.681059 8404 raft_consensus.cc:729] T 9e71fe92ef174285966b215575c333fd P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9fc1f19fe2a4e4f93eddca1b80f3fb4, State: Initialized, Role: FOLLOWER
I20260620 00:07:15.681110 8404 consensus_queue.cc:260] T 9e71fe92ef174285966b215575c333fd P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } }
I20260620 00:07:15.681214 8404 ts_tablet_manager.cc:1428] T 9e71fe92ef174285966b215575c333fd P d9fc1f19fe2a4e4f93eddca1b80f3fb4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 00:07:15.910315 8457 raft_consensus.cc:482] T 9e71fe92ef174285966b215575c333fd P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:07:15.910424 8457 raft_consensus.cc:504] T 9e71fe92ef174285966b215575c333fd P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } }
I20260620 00:07:15.910616 8457 leader_election.cc:288] T 9e71fe92ef174285966b215575c333fd P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f34094d8e684686b4004cf910f8f16a (127.0.98.1:35827), 69ca88ea258443a39dad68029d288f95 (127.0.98.2:34395)
I20260620 00:07:15.910872 8049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e71fe92ef174285966b215575c333fd" candidate_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f34094d8e684686b4004cf910f8f16a" is_pre_election: true
I20260620 00:07:15.910871 8189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e71fe92ef174285966b215575c333fd" candidate_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ca88ea258443a39dad68029d288f95" is_pre_election: true
I20260620 00:07:15.910980 8189 raft_consensus.cc:2454] T 9e71fe92ef174285966b215575c333fd P 69ca88ea258443a39dad68029d288f95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9fc1f19fe2a4e4f93eddca1b80f3fb4 in term 0.
I20260620 00:07:15.910981 8049 raft_consensus.cc:2454] T 9e71fe92ef174285966b215575c333fd P 6f34094d8e684686b4004cf910f8f16a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9fc1f19fe2a4e4f93eddca1b80f3fb4 in term 0.
I20260620 00:07:15.911170 8263 leader_election.cc:302] T 9e71fe92ef174285966b215575c333fd P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ca88ea258443a39dad68029d288f95, d9fc1f19fe2a4e4f93eddca1b80f3fb4; no voters:
I20260620 00:07:15.911273 8457 raft_consensus.cc:2781] T 9e71fe92ef174285966b215575c333fd P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:07:15.911304 8457 raft_consensus.cc:482] T 9e71fe92ef174285966b215575c333fd P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:07:15.911350 8457 raft_consensus.cc:3037] T 9e71fe92ef174285966b215575c333fd P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:15.912129 8457 raft_consensus.cc:504] T 9e71fe92ef174285966b215575c333fd P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } }
I20260620 00:07:15.912292 8457 leader_election.cc:288] T 9e71fe92ef174285966b215575c333fd P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [CANDIDATE]: Term 1 election: Requested vote from peers 6f34094d8e684686b4004cf910f8f16a (127.0.98.1:35827), 69ca88ea258443a39dad68029d288f95 (127.0.98.2:34395)
I20260620 00:07:15.912470 8189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e71fe92ef174285966b215575c333fd" candidate_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ca88ea258443a39dad68029d288f95"
I20260620 00:07:15.912474 8049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e71fe92ef174285966b215575c333fd" candidate_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f34094d8e684686b4004cf910f8f16a"
I20260620 00:07:15.912560 8189 raft_consensus.cc:3037] T 9e71fe92ef174285966b215575c333fd P 69ca88ea258443a39dad68029d288f95 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:15.912560 8049 raft_consensus.cc:3037] T 9e71fe92ef174285966b215575c333fd P 6f34094d8e684686b4004cf910f8f16a [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:15.913389 8049 raft_consensus.cc:2454] T 9e71fe92ef174285966b215575c333fd P 6f34094d8e684686b4004cf910f8f16a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9fc1f19fe2a4e4f93eddca1b80f3fb4 in term 1.
I20260620 00:07:15.913403 8189 raft_consensus.cc:2454] T 9e71fe92ef174285966b215575c333fd P 69ca88ea258443a39dad68029d288f95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9fc1f19fe2a4e4f93eddca1b80f3fb4 in term 1.
I20260620 00:07:15.913602 8263 leader_election.cc:302] T 9e71fe92ef174285966b215575c333fd P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ca88ea258443a39dad68029d288f95, d9fc1f19fe2a4e4f93eddca1b80f3fb4; no voters:
I20260620 00:07:15.913743 8457 raft_consensus.cc:2781] T 9e71fe92ef174285966b215575c333fd P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:07:15.913874 8457 raft_consensus.cc:686] T 9e71fe92ef174285966b215575c333fd P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 1 LEADER]: Becoming Leader. State: Replica: d9fc1f19fe2a4e4f93eddca1b80f3fb4, State: Running, Role: LEADER
I20260620 00:07:15.913923 8457 consensus_queue.cc:237] T 9e71fe92ef174285966b215575c333fd P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } }
I20260620 00:07:15.914625 7891 catalog_manager.cc:5495] T 9e71fe92ef174285966b215575c333fd P d9fc1f19fe2a4e4f93eddca1b80f3fb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to d9fc1f19fe2a4e4f93eddca1b80f3fb4 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9fc1f19fe2a4e4f93eddca1b80f3fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42519 } health_report { overall_health: HEALTHY } } }
I20260620 00:07:15.969209 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:15.966811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44318 (local address 127.0.98.3:42519)
0620 00:07:15.966984 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:15.966986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:15.966993 (+ 7us) server_negotiation.cc:407] Connection header received
0620 00:07:15.967012 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:15.967013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:15.967023 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:15.967081 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:07:15.967169 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:15.967655 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:15.968458 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:15.968664 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:15.968746 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:15.968936 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:07:15.968966 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:07:15.969008 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:15.969065 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 00:07:15.969084 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":109,"threads_started":1}
I20260620 00:07:15.973845 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:15.971563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53227 (local address 127.0.98.3:42519)
0620 00:07:15.971622 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:15.971624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:15.971728 (+ 104us) server_negotiation.cc:407] Connection header received
0620 00:07:15.971794 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:15.971794 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:15.971804 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:15.971850 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:07:15.971941 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:15.972389 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:15.973097 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:15.973280 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:15.973334 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:15.973578 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:07:15.973603 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:07:15.973644 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:15.973697 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 00:07:15.973721 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":42}
I20260620 00:07:15.973850 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:15.971499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42519 (local address 127.0.98.3:53227)
0620 00:07:15.971665 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:15.971672 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:15.971721 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:15.971853 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:15.971853 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:15.971854 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 00:07:15.971891 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:15.971892 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:15.972460 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:15.972461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:15.973021 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:15.973023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:15.973457 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:15.973458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:15.973503 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:15.973507 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 00:07:15.973659 (+ 152us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 00:07:15.973661 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 00:07:15.973696 (+ 35us) client_negotiation.cc:240] Negotiation successful
0620 00:07:15.973767 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":3,"thread_start_us":76,"threads_started":1}
I20260620 00:07:15.975934 8189 raft_consensus.cc:1261] T 9e71fe92ef174285966b215575c333fd P 69ca88ea258443a39dad68029d288f95 [term 1 FOLLOWER]: Refusing update from remote peer d9fc1f19fe2a4e4f93eddca1b80f3fb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:07:15.975972 8049 raft_consensus.cc:1261] T 9e71fe92ef174285966b215575c333fd P 6f34094d8e684686b4004cf910f8f16a [term 1 FOLLOWER]: Refusing update from remote peer d9fc1f19fe2a4e4f93eddca1b80f3fb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:07:15.976153 8499 consensus_queue.cc:1035] T 9e71fe92ef174285966b215575c333fd P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ca88ea258443a39dad68029d288f95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:07:15.976270 8497 consensus_queue.cc:1035] T 9e71fe92ef174285966b215575c333fd P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f34094d8e684686b4004cf910f8f16a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:07:15.986915 7892 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:07:15.987038 7892 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:07:15.987723 7892 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05cccc5116e84b959eee49c8c6ca05a4: Sending DeleteTablet for 3 replicas of tablet 9e71fe92ef174285966b215575c333fd
I20260620 00:07:15.988013 8169 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e71fe92ef174285966b215575c333fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:07:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47518
I20260620 00:07:15.988047 8029 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e71fe92ef174285966b215575c333fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:07:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35558
I20260620 00:07:15.988118 8492 tablet_replica.cc:330] T 9e71fe92ef174285966b215575c333fd P 69ca88ea258443a39dad68029d288f95: stopping tablet replica
I20260620 00:07:15.988148 8427 tablet_replica.cc:330] T 9e71fe92ef174285966b215575c333fd P 6f34094d8e684686b4004cf910f8f16a: stopping tablet replica
I20260620 00:07:15.988169 8492 raft_consensus.cc:2229] T 9e71fe92ef174285966b215575c333fd P 69ca88ea258443a39dad68029d288f95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:07:15.988194 8427 raft_consensus.cc:2229] T 9e71fe92ef174285966b215575c333fd P 6f34094d8e684686b4004cf910f8f16a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:07:15.988207 8492 raft_consensus.cc:2258] T 9e71fe92ef174285966b215575c333fd P 69ca88ea258443a39dad68029d288f95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:07:15.988231 8427 raft_consensus.cc:2258] T 9e71fe92ef174285966b215575c333fd P 6f34094d8e684686b4004cf910f8f16a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:07:15.988471 8308 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e71fe92ef174285966b215575c333fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:07:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44294
I20260620 00:07:15.988559 8493 tablet_replica.cc:330] T 9e71fe92ef174285966b215575c333fd P d9fc1f19fe2a4e4f93eddca1b80f3fb4: stopping tablet replica
I20260620 00:07:15.988571 8427 ts_tablet_manager.cc:1905] T 9e71fe92ef174285966b215575c333fd P 6f34094d8e684686b4004cf910f8f16a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:07:15.988605 8493 raft_consensus.cc:2229] T 9e71fe92ef174285966b215575c333fd P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:07:15.988721 8493 raft_consensus.cc:2258] T 9e71fe92ef174285966b215575c333fd P d9fc1f19fe2a4e4f93eddca1b80f3fb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:07:15.989108 8492 ts_tablet_manager.cc:1905] T 9e71fe92ef174285966b215575c333fd P 69ca88ea258443a39dad68029d288f95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:07:15.990793 8493 ts_tablet_manager.cc:1905] T 9e71fe92ef174285966b215575c333fd P d9fc1f19fe2a4e4f93eddca1b80f3fb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:07:15.991050 8492 ts_tablet_manager.cc:1918] T 9e71fe92ef174285966b215575c333fd P 69ca88ea258443a39dad68029d288f95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 00:07:15.991119 8492 log.cc:1196] T 9e71fe92ef174285966b215575c333fd P 69ca88ea258443a39dad68029d288f95: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-1/wal/wals/9e71fe92ef174285966b215575c333fd
I20260620 00:07:15.991334 8492 ts_tablet_manager.cc:1939] T 9e71fe92ef174285966b215575c333fd P 69ca88ea258443a39dad68029d288f95: Deleting consensus metadata
I20260620 00:07:15.991758 7879 catalog_manager.cc:4841] TS 69ca88ea258443a39dad68029d288f95 (127.0.98.2:34395): tablet 9e71fe92ef174285966b215575c333fd (table test-table [id=6103af9cab9e4e98bee063136a830010]) successfully deleted
I20260620 00:07:15.992360 8493 ts_tablet_manager.cc:1918] T 9e71fe92ef174285966b215575c333fd P d9fc1f19fe2a4e4f93eddca1b80f3fb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 00:07:15.992430 8493 log.cc:1196] T 9e71fe92ef174285966b215575c333fd P d9fc1f19fe2a4e4f93eddca1b80f3fb4: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914001714546-392-0/minicluster-data/ts-2/wal/wals/9e71fe92ef174285966b215575c333fd
I20260620 00:07:15.992635 8493 ts_tablet_manager.cc:1939] T 9e71fe92ef174285966b215575c333fd P d9fc1f19fe2a4e4f93eddca1b80f3fb4: Deleting consensus metadata
I20260620 00:07:15.993038 7878 catalog_manager.cc:4841] TS d9fc1f19fe2a4e4f93eddca1b80f3fb4 (127.0.98.3:42519): tablet 9e71fe92ef174285966b215575c333fd (table test-table [id=6103af9cab9e4e98bee063136a830010]) successfully deleted
I20260620 00:07:15.993320 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 7964
W20260620 00:07:16.002293 7877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35827 (error 108)
I20260620 00:07:16.002595 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 8104
I20260620 00:07:16.002897 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:16.002705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35827 (local address 127.0.0.1:35600)
0620 00:07:16.002797 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:16.002841 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":39,"threads_started":1}
W20260620 00:07:16.003088 7877 catalog_manager.cc:4584] TS 6f34094d8e684686b4004cf910f8f16a (127.0.98.1:35827): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9e71fe92ef174285966b215575c333fd: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35827: connect: Connection refused (error 111)
I20260620 00:07:16.010715 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 8244
I20260620 00:07:16.017729 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 7860
2026-06-20T00:07:16Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3323 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:07:16 dist-test-slave-grqt krb5kdc[8512](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:07:16 dist-test-slave-grqt krb5kdc[8512](info): set up 2 sockets
Jun 20 00:07:16 dist-test-slave-grqt krb5kdc[8512](info): commencing operation
krb5kdc: starting...
W20260620 00:07:17.082197 392 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:07:17 dist-test-slave-grqt krb5kdc[8512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:07:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:07:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:07:17.209157 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34381
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:07:17.294040 8525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:17.294306 8525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:17.294332 8525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:17.295620 8525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:07:17.295676 8525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:17.295692 8525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:17.295706 8525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:07:17.295717 8525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:07:17.297426 8525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:17.297716 8525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:17.297986 8525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:17.303093 8533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:17.303093 8530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:17.303113 8531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:17.303453 8525 server_base.cc:1034] running on GCE node
I20260620 00:07:17.303751 8525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:17.304067 8525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:17.305220 8525 hybrid_clock.cc:648] HybridClock initialized: now 1781914037305160 us; error 78 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T00:07:17Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1288 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:07:17 dist-test-slave-grqt krb5kdc[8538](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:07:17 dist-test-slave-grqt krb5kdc[8538](info): set up 2 sockets
Jun 20 00:07:17 dist-test-slave-grqt krb5kdc[8538](info): commencing operation
krb5kdc: starting...
W20260620 00:07:18.382225 392 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:07:18 dist-test-slave-grqt krb5kdc[8538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:07:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:07:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:07:18.515957 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37849
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37849
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:07:18.608724 8551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:18.608990 8551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:18.609015 8551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:18.610424 8551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:07:18.610483 8551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:18.610499 8551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:18.610512 8551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:07:18.610524 8551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:07:18.612318 8551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:18.612622 8551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:18.612898 8551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:18.618304 8557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:18.618469 8556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:18.618355 8551 server_base.cc:1034] running on GCE node
W20260620 00:07:18.618304 8559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:18.618872 8551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:18.619239 8551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:18.620432 8551 hybrid_clock.cc:648] HybridClock initialized: now 1781914038620398 us; error 56 us; skew 500 ppm
Jun 20 00:07:18 dist-test-slave-grqt krb5kdc[8538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:18.623370 8551 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:07:18.624107 8551 webserver.cc:460] Webserver started at http://127.0.98.62:35985/ using document root <none> and password file <none>
I20260620 00:07:18.624385 8551 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:18.624464 8551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:18.624621 8551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:18.625977 8551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "6e28508195e74cd885d791a647ba2f5e"
format_stamp: "Formatted at 2026-06-20 00:07:18 on dist-test-slave-grqt"
server_key: "b7b5a77d758710901f8e045b787f8528"
server_key_iv: "d25887426aa2a53495678c855ef1708a"
server_key_version: "encryptionkey@0"
I20260620 00:07:18.626459 8551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "6e28508195e74cd885d791a647ba2f5e"
format_stamp: "Formatted at 2026-06-20 00:07:18 on dist-test-slave-grqt"
server_key: "b7b5a77d758710901f8e045b787f8528"
server_key_iv: "d25887426aa2a53495678c855ef1708a"
server_key_version: "encryptionkey@0"
I20260620 00:07:18.628334 8551 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 00:07:18.629618 8566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:18.629892 8551 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:18.630007 8551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "6e28508195e74cd885d791a647ba2f5e"
format_stamp: "Formatted at 2026-06-20 00:07:18 on dist-test-slave-grqt"
server_key: "b7b5a77d758710901f8e045b787f8528"
server_key_iv: "d25887426aa2a53495678c855ef1708a"
server_key_version: "encryptionkey@0"
I20260620 00:07:18.630117 8551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:18.644635 8551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:18.647694 8551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:18.647933 8551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:18.653015 8551 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37849
I20260620 00:07:18.653106 8618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37849 every 8 connection(s)
I20260620 00:07:18.653754 8551 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:07:18.654575 8619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:18.656946 8619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e: Bootstrap starting.
I20260620 00:07:18.657625 8619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:18.657953 8619 log.cc:824] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e: Log is configured to *not* fsync() on all Append() calls
I20260620 00:07:18.658743 8619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e: No bootstrap required, opened a new log
I20260620 00:07:18.660620 8619 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e28508195e74cd885d791a647ba2f5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37849 } }
I20260620 00:07:18.660804 8619 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:18.660830 8619 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e28508195e74cd885d791a647ba2f5e, State: Initialized, Role: FOLLOWER
I20260620 00:07:18.660990 8619 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e28508195e74cd885d791a647ba2f5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37849 } }
I20260620 00:07:18.661075 8619 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:07:18.661101 8619 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:07:18.661137 8619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:18.661662 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 8551
I20260620 00:07:18.661767 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:07:18.662017 392 external_mini_cluster.cc:1428] Setting key 9d9f8d575fad3aba35a42e715255af02
I20260620 00:07:18.662564 8619 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e28508195e74cd885d791a647ba2f5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37849 } }
I20260620 00:07:18.662698 8619 leader_election.cc:302] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e28508195e74cd885d791a647ba2f5e; no voters:
I20260620 00:07:18.662945 8619 leader_election.cc:288] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:07:18.662981 8624 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:07:18.663094 8624 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e [term 1 LEADER]: Becoming Leader. State: Replica: 6e28508195e74cd885d791a647ba2f5e, State: Running, Role: LEADER
I20260620 00:07:18.663254 8624 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e28508195e74cd885d791a647ba2f5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37849 } }
I20260620 00:07:18.663264 8619 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:07:18.665120 8625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e28508195e74cd885d791a647ba2f5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e28508195e74cd885d791a647ba2f5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37849 } } }
I20260620 00:07:18.665149 8626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e28508195e74cd885d791a647ba2f5e. Latest consensus state: current_term: 1 leader_uuid: "6e28508195e74cd885d791a647ba2f5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e28508195e74cd885d791a647ba2f5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37849 } } }
I20260620 00:07:18.665290 8625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e [sys.catalog]: This master's current role is: LEADER
I20260620 00:07:18.665297 8626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e [sys.catalog]: This master's current role is: LEADER
W20260620 00:07:18.665391 8639 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:07:18.665467 8639 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:07:18.665536 8641 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:07:18.665735 8641 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:07:18.667348 8641 catalog_manager.cc:1347] Generated new cluster ID: 5ecefc6ac8ea4f75a7c45df909c44832
I20260620 00:07:18.667408 8641 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 00:07:18 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:07:18.676802 8623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:18.662643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34746 (local address 127.0.98.62:37849)
0620 00:07:18.662862 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:18.662886 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:18.662918 (+ 32us) server_negotiation.cc:407] Connection header received
0620 00:07:18.663077 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:18.663086 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:18.663155 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:18.664158 (+ 1003us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:18.664316 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:18.666495 (+ 2179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:18.667603 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:18.668286 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:18.668406 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:18.671896 (+ 3490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:18.671902 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:18.671902 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:18.671917 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:18.674694 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:18.675291 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:18.675295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:18.675296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:18.675441 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:18.675708 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:18.675709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:18.675710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:18.676072 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:18.676230 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:18.676388 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 00:07:18.676548 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":104,"threads_started":1}
I20260620 00:07:18.697183 8641 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:07:18.697822 8641 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:07:18.705350 8641 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e: Generated new TSK 0
I20260620 00:07:18.705641 8641 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:07:18.740690 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37849
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:07:18.823134 8645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:18.823364 8645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:18.823410 8645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:18.824553 8645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:18.824625 8645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:18.824671 8645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:07:18.826288 8645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:18.826565 8645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:18.826802 8645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:18.832671 8651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:18.832688 8653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:18.832824 8645 server_base.cc:1034] running on GCE node
W20260620 00:07:18.832674 8650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:18.833154 8645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:18.833521 8645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:18.834720 8645 hybrid_clock.cc:648] HybridClock initialized: now 1781914038834685 us; error 58 us; skew 500 ppm
Jun 20 00:07:18 dist-test-slave-grqt krb5kdc[8538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:18.837415 8645 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:07:18.837976 8645 webserver.cc:460] Webserver started at http://127.0.98.1:36749/ using document root <none> and password file <none>
I20260620 00:07:18.838240 8645 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:18.838312 8645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:18.838466 8645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:18.839564 8645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-0/data/instance:
uuid: "033d30f20a4b4530be37de6bd53d5752"
format_stamp: "Formatted at 2026-06-20 00:07:18 on dist-test-slave-grqt"
server_key: "da252a4fc26548d7e42fc1c3e0642920"
server_key_iv: "d9b2672c594a649d4f74cab501ba1f60"
server_key_version: "encryptionkey@0"
I20260620 00:07:18.839937 8645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-0/wal/instance:
uuid: "033d30f20a4b4530be37de6bd53d5752"
format_stamp: "Formatted at 2026-06-20 00:07:18 on dist-test-slave-grqt"
server_key: "da252a4fc26548d7e42fc1c3e0642920"
server_key_iv: "d9b2672c594a649d4f74cab501ba1f60"
server_key_version: "encryptionkey@0"
I20260620 00:07:18.841338 8645 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:07:18.842188 8660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:18.842465 8645 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:18.842595 8645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-0/wal
uuid: "033d30f20a4b4530be37de6bd53d5752"
format_stamp: "Formatted at 2026-06-20 00:07:18 on dist-test-slave-grqt"
server_key: "da252a4fc26548d7e42fc1c3e0642920"
server_key_iv: "d9b2672c594a649d4f74cab501ba1f60"
server_key_version: "encryptionkey@0"
I20260620 00:07:18.842686 8645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:18.869948 8645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:18.872231 8645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:18.872435 8645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:18.872700 8645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:07:18.873023 8645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:07:18.873083 8645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:18.873121 8645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:07:18.873168 8645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:18.879355 8645 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40735
I20260620 00:07:18.879390 8773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40735 every 8 connection(s)
I20260620 00:07:18.879724 8645 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:07:18.886174 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 8645
Jun 20 00:07:18 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:07:18.886267 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:07:18.886507 392 external_mini_cluster.cc:1428] Setting key f00f0065e84f62fdce05ebe9ca4e030a
I20260620 00:07:18.890460 8623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:18.880526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59377 (local address 127.0.98.62:37849)
0620 00:07:18.880586 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:18.880588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:18.880807 (+ 219us) server_negotiation.cc:407] Connection header received
0620 00:07:18.881880 (+ 1073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:18.881882 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:18.881897 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:18.881954 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:18.882608 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:18.883124 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:18.884140 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:18.884360 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:18.884451 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:18.886922 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:18.886924 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:18.886924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:18.886933 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:18.888874 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:18.889311 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:18.889312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:18.889313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:18.889415 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:18.889741 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:18.889741 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:18.889742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:18.889846 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:18.889899 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:18.890320 (+ 421us) server_negotiation.cc:299] Negotiation successful
0620 00:07:18.890393 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 00:07:18.890781 8776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:18.880489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.1:59377)
0620 00:07:18.880680 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:18.880721 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:18.881759 (+ 1038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:18.882029 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:18.882032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:18.882299 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:18.882503 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:18.882508 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:18.883227 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:18.883228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:18.884040 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:18.884043 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:18.884461 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:18.884462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:18.884583 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:18.885104 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:18.885112 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:18.886790 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:18.888979 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:18.888986 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:18.888988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:18.889226 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:18.889484 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:18.889485 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:18.889486 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:18.889660 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:18.889946 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:18.889952 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:18.890126 (+ 174us) client_negotiation.cc:769] Sending connection context
0620 00:07:18.890288 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 00:07:18.890477 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":96,"threads_started":1}
I20260620 00:07:18.891424 8774 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37849
I20260620 00:07:18.891542 8774 heartbeater.cc:464] Registering TS with master...
I20260620 00:07:18.891741 8774 heartbeater.cc:511] Master 127.0.98.62:37849 requested a full tablet report, sending...
I20260620 00:07:18.892324 8583 ts_manager.cc:194] Registered new tserver with Master: 033d30f20a4b4530be37de6bd53d5752 (127.0.98.1:40735)
I20260620 00:07:18.892963 8583 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59377
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:07:18.916918 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37849
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:07:18.998802 8779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:18.999070 8779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:18.999109 8779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:19.000309 8779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:19.000392 8779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:19.000439 8779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:07:19.002065 8779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:19.002353 8779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:19.002604 8779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:19.007922 8787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:19.007952 8785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:19.007968 8784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:19.008256 8779 server_base.cc:1034] running on GCE node
I20260620 00:07:19.008481 8779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:19.008697 8779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:19.009907 8779 hybrid_clock.cc:648] HybridClock initialized: now 1781914039009894 us; error 32 us; skew 500 ppm
Jun 20 00:07:19 dist-test-slave-grqt krb5kdc[8538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914039, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:19.012758 8779 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:07:19.013231 8779 webserver.cc:460] Webserver started at http://127.0.98.2:37799/ using document root <none> and password file <none>
I20260620 00:07:19.013550 8779 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:19.013625 8779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:19.013782 8779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:19.014999 8779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-1/data/instance:
uuid: "cd2fa9e037c44ccfafef734569a0e5fb"
format_stamp: "Formatted at 2026-06-20 00:07:19 on dist-test-slave-grqt"
server_key: "f2d34848c5af683ef4a1abfb4bb05094"
server_key_iv: "f67ac052505fd98f1dec21540fea9dab"
server_key_version: "encryptionkey@0"
I20260620 00:07:19.015379 8779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cd2fa9e037c44ccfafef734569a0e5fb"
format_stamp: "Formatted at 2026-06-20 00:07:19 on dist-test-slave-grqt"
server_key: "f2d34848c5af683ef4a1abfb4bb05094"
server_key_iv: "f67ac052505fd98f1dec21540fea9dab"
server_key_version: "encryptionkey@0"
I20260620 00:07:19.016834 8779 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:07:19.017814 8794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:19.018034 8779 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:19.018143 8779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-1/wal
uuid: "cd2fa9e037c44ccfafef734569a0e5fb"
format_stamp: "Formatted at 2026-06-20 00:07:19 on dist-test-slave-grqt"
server_key: "f2d34848c5af683ef4a1abfb4bb05094"
server_key_iv: "f67ac052505fd98f1dec21540fea9dab"
server_key_version: "encryptionkey@0"
I20260620 00:07:19.018221 8779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:19.031415 8779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:19.034065 8779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:19.034258 8779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:19.034533 8779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:07:19.034906 8779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:07:19.034965 8779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:19.035009 8779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:07:19.035037 8779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:19.041082 8779 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38743
I20260620 00:07:19.041124 8907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38743 every 8 connection(s)
I20260620 00:07:19.041563 8779 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:07:19.042151 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 8779
I20260620 00:07:19.042243 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:07:19.042404 392 external_mini_cluster.cc:1428] Setting key d8f96262ef854214de8b81d1619a7abe
Jun 20 00:07:19 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914039, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 00:07:19.054011 8623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:19.043145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57771 (local address 127.0.98.62:37849)
0620 00:07:19.043299 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:19.043301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:19.043483 (+ 182us) server_negotiation.cc:407] Connection header received
0620 00:07:19.044965 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:19.044966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:19.044983 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:19.045085 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:19.045960 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.046637 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.047698 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.047906 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.048000 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:19.050580 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:19.050582 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:19.050582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:19.050590 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:19.052275 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:19.052770 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:19.052772 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:19.052772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:19.052835 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:19.053188 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:19.053189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:19.053189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:19.053323 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:19.053424 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:19.053818 (+ 394us) server_negotiation.cc:299] Negotiation successful
0620 00:07:19.053914 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260620 00:07:19.054204 8910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:19.043120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.2:57771)
0620 00:07:19.043330 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:19.043404 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:19.044801 (+ 1397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:19.045180 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:19.045184 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:19.045623 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:19.045855 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:19.045860 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.046723 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.046725 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:19.047593 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:19.047596 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.048012 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.048013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:19.048149 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:19.048742 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:19.048757 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:19.050461 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:19.052401 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:19.052409 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:19.052412 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:19.052664 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:19.052911 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:19.052911 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:19.052912 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:19.053103 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:19.053466 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:19.053484 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:19.053654 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 00:07:19.053782 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 00:07:19.053954 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":109,"threads_started":1}
I20260620 00:07:19.054704 8908 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37849
I20260620 00:07:19.054849 8908 heartbeater.cc:464] Registering TS with master...
I20260620 00:07:19.055060 8908 heartbeater.cc:511] Master 127.0.98.62:37849 requested a full tablet report, sending...
I20260620 00:07:19.055568 8583 ts_manager.cc:194] Registered new tserver with Master: cd2fa9e037c44ccfafef734569a0e5fb (127.0.98.2:38743)
I20260620 00:07:19.056170 8583 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57771
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:07:19.069471 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37849
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:07:19.156883 8913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:19.157222 8913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:19.157274 8913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:19.158797 8913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:19.158908 8913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:19.158960 8913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:07:19.160862 8913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:19.161207 8913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:19.161512 8913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:19.167937 8921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:19.168093 8913 server_base.cc:1034] running on GCE node
W20260620 00:07:19.168174 8919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:19.167995 8918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:19.168443 8913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:19.168742 8913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:19.169945 8913 hybrid_clock.cc:648] HybridClock initialized: now 1781914039169937 us; error 71 us; skew 500 ppm
Jun 20 00:07:19 dist-test-slave-grqt krb5kdc[8538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914039, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:19.172890 8913 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:07:19.173512 8913 webserver.cc:460] Webserver started at http://127.0.98.3:34609/ using document root <none> and password file <none>
I20260620 00:07:19.173764 8913 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:19.173849 8913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:19.174006 8913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:19.175213 8913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-2/data/instance:
uuid: "4de626faa66544378c59d7e6ad5d9553"
format_stamp: "Formatted at 2026-06-20 00:07:19 on dist-test-slave-grqt"
server_key: "b95afc73d7fe29160f3b059e8dc5b22a"
server_key_iv: "fc3afc0b7cf6abfd082cca41a37412d4"
server_key_version: "encryptionkey@0"
I20260620 00:07:19.175645 8913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4de626faa66544378c59d7e6ad5d9553"
format_stamp: "Formatted at 2026-06-20 00:07:19 on dist-test-slave-grqt"
server_key: "b95afc73d7fe29160f3b059e8dc5b22a"
server_key_iv: "fc3afc0b7cf6abfd082cca41a37412d4"
server_key_version: "encryptionkey@0"
I20260620 00:07:19.177239 8913 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:07:19.178131 8928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:19.178488 8913 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:07:19.178628 8913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-2/wal
uuid: "4de626faa66544378c59d7e6ad5d9553"
format_stamp: "Formatted at 2026-06-20 00:07:19 on dist-test-slave-grqt"
server_key: "b95afc73d7fe29160f3b059e8dc5b22a"
server_key_iv: "fc3afc0b7cf6abfd082cca41a37412d4"
server_key_version: "encryptionkey@0"
I20260620 00:07:19.178726 8913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:19.199164 8913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:19.202090 8913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:19.202298 8913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:19.202611 8913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:07:19.202970 8913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:07:19.203030 8913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:19.203073 8913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:07:19.203104 8913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:19.210085 8913 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41411
I20260620 00:07:19.210223 9041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41411 every 8 connection(s)
I20260620 00:07:19.210613 8913 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:07:19.215081 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 8913
I20260620 00:07:19.215191 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:07:19.215420 392 external_mini_cluster.cc:1428] Setting key 9370d659fdd4033c25112fb4a7ef9800
Jun 20 00:07:19 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914039, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:07:19.225281 9044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:19.211514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.3:33353)
0620 00:07:19.211694 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:19.211741 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:19.212813 (+ 1072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:19.213141 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:19.213145 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:19.213600 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:19.213847 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:19.213853 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.214778 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.214779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:19.216465 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:19.216470 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.217198 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.217200 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:19.217597 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:19.218381 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:19.218391 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:19.220360 (+ 1969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:19.222894 (+ 2534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:19.222912 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:19.222915 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:19.223445 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:19.223942 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:19.223943 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:19.223943 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:19.224102 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:19.224520 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:19.224528 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:19.224694 (+ 166us) client_negotiation.cc:769] Sending connection context
0620 00:07:19.224813 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 00:07:19.224985 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":102,"threads_started":1}
I20260620 00:07:19.225322 8623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:19.211548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33353 (local address 127.0.98.62:37849)
0620 00:07:19.211641 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:19.211643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:19.211885 (+ 242us) server_negotiation.cc:407] Connection header received
0620 00:07:19.212960 (+ 1075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:19.212961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:19.212976 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:19.213069 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:19.213976 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.214605 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.216635 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.216975 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.217045 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:19.220741 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:19.220743 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:19.220743 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:19.220752 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:19.222726 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:19.223634 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:19.223636 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:19.223637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:19.223716 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:19.224207 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:19.224208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:19.224208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:19.224408 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:19.224481 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:19.224969 (+ 488us) server_negotiation.cc:299] Negotiation successful
0620 00:07:19.225136 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 00:07:19.226271 9042 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37849
I20260620 00:07:19.226398 9042 heartbeater.cc:464] Registering TS with master...
I20260620 00:07:19.226680 9042 heartbeater.cc:511] Master 127.0.98.62:37849 requested a full tablet report, sending...
I20260620 00:07:19.227252 8583 ts_manager.cc:194] Registered new tserver with Master: 4de626faa66544378c59d7e6ad5d9553 (127.0.98.3:41411)
I20260620 00:07:19.227851 8583 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33353
I20260620 00:07:19.228349 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 00:07:19.321651 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 8645
I20260620 00:07:19.326543 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 8779
I20260620 00:07:19.331241 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 8913
I20260620 00:07:19.336004 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 8551
I20260620 00:07:19.341311 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37849
--webserver_interface=127.0.98.62
--webserver_port=35985
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37849
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:07:19.426628 9051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:19.426856 9051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:19.426882 9051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:19.428203 9051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:07:19.428295 9051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:19.428323 9051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:19.428354 9051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:07:19.428368 9051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:07:19.430047 9051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:19.430274 9051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:19.430480 9051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:19.436359 9057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:19.436337 9056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:19.436343 9059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:19.436654 9051 server_base.cc:1034] running on GCE node
I20260620 00:07:19.436872 9051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:19.437136 9051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:19.438369 9051 hybrid_clock.cc:648] HybridClock initialized: now 1781914039438355 us; error 31 us; skew 500 ppm
Jun 20 00:07:19 dist-test-slave-grqt krb5kdc[8538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914039, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:19.441144 9051 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:07:19.441730 9051 webserver.cc:460] Webserver started at http://127.0.98.62:35985/ using document root <none> and password file <none>
I20260620 00:07:19.442045 9051 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:19.442117 9051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:19.444024 9051 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:07:19.445030 9066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:19.445515 9051 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:19.445633 9051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "6e28508195e74cd885d791a647ba2f5e"
format_stamp: "Formatted at 2026-06-20 00:07:18 on dist-test-slave-grqt"
server_key: "b7b5a77d758710901f8e045b787f8528"
server_key_iv: "d25887426aa2a53495678c855ef1708a"
server_key_version: "encryptionkey@0"
I20260620 00:07:19.446102 9051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:19.459132 9051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:19.462515 9051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:19.462675 9051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:19.467454 9051 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37849
I20260620 00:07:19.467555 9118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37849 every 8 connection(s)
I20260620 00:07:19.468178 9051 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:07:19.470028 9119 sys_catalog.cc:263] Verifying existing consensus state
I20260620 00:07:19.470803 9119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e: Bootstrap starting.
I20260620 00:07:19.476891 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 9051
I20260620 00:07:19.477234 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40735
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37849
--webserver_port=36749
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:07:19.480304 9119 log.cc:824] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e: Log is configured to *not* fsync() on all Append() calls
I20260620 00:07:19.482287 9119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 00:07:19.482607 9119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e: Bootstrap complete.
I20260620 00:07:19.484941 9119 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e28508195e74cd885d791a647ba2f5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37849 } }
I20260620 00:07:19.485231 9119 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e28508195e74cd885d791a647ba2f5e, State: Initialized, Role: FOLLOWER
I20260620 00:07:19.485514 9119 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e28508195e74cd885d791a647ba2f5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37849 } }
I20260620 00:07:19.485632 9119 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:07:19.485690 9119 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:07:19.485741 9119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e [term 1 FOLLOWER]: Advancing to term 2
I20260620 00:07:19.486814 9119 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e28508195e74cd885d791a647ba2f5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37849 } }
I20260620 00:07:19.486953 9119 leader_election.cc:302] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e28508195e74cd885d791a647ba2f5e; no voters:
I20260620 00:07:19.487202 9119 leader_election.cc:288] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 00:07:19.487236 9123 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e [term 2 FOLLOWER]: Leader election won for term 2
I20260620 00:07:19.487350 9123 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e [term 2 LEADER]: Becoming Leader. State: Replica: 6e28508195e74cd885d791a647ba2f5e, State: Running, Role: LEADER
I20260620 00:07:19.487449 9123 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e28508195e74cd885d791a647ba2f5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37849 } }
I20260620 00:07:19.487488 9119 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:07:19.487779 9124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e28508195e74cd885d791a647ba2f5e. Latest consensus state: current_term: 2 leader_uuid: "6e28508195e74cd885d791a647ba2f5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e28508195e74cd885d791a647ba2f5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37849 } } }
I20260620 00:07:19.487905 9124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e [sys.catalog]: This master's current role is: LEADER
I20260620 00:07:19.487994 9125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6e28508195e74cd885d791a647ba2f5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e28508195e74cd885d791a647ba2f5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37849 } } }
I20260620 00:07:19.488153 9125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e [sys.catalog]: This master's current role is: LEADER
I20260620 00:07:19.488427 9132 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:07:19.489056 9132 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:07:19.489259 9132 catalog_manager.cc:1259] Loaded cluster ID: 5ecefc6ac8ea4f75a7c45df909c44832
I20260620 00:07:19.489528 9132 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:07:19.490697 9132 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:07:19.490880 9132 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6e28508195e74cd885d791a647ba2f5e: Loaded TSK: 0
I20260620 00:07:19.491103 9132 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 00:07:19.567652 9121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:19.567946 9121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:19.567991 9121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:19.569394 9121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:19.569486 9121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:19.569551 9121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:07:19.571420 9121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40735
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36749
--tserver_master_addrs=127.0.98.62:37849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:19.571753 9121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:19.572280 9121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:19.578493 9145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:19.578871 9148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:19.579098 9146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:19.579380 9121 server_base.cc:1034] running on GCE node
I20260620 00:07:19.579607 9121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:19.579967 9121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:19.581185 9121 hybrid_clock.cc:648] HybridClock initialized: now 1781914039581149 us; error 50 us; skew 500 ppm
Jun 20 00:07:19 dist-test-slave-grqt krb5kdc[8538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914039, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:19.588850 9121 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 00:07:19.589617 9121 webserver.cc:460] Webserver started at http://127.0.98.1:36749/ using document root <none> and password file <none>
I20260620 00:07:19.589946 9121 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:19.590030 9121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:19.592157 9121 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 00:07:19.593024 9155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:19.593349 9121 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:07:19.593492 9121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-0/wal
uuid: "033d30f20a4b4530be37de6bd53d5752"
format_stamp: "Formatted at 2026-06-20 00:07:18 on dist-test-slave-grqt"
server_key: "da252a4fc26548d7e42fc1c3e0642920"
server_key_iv: "d9b2672c594a649d4f74cab501ba1f60"
server_key_version: "encryptionkey@0"
I20260620 00:07:19.594018 9121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:19.619415 9121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:19.622465 9121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:19.622635 9121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:19.622932 9121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:07:19.623273 9121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:07:19.623318 9121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:19.623349 9121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:07:19.623368 9121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:19.632289 9121 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40735
I20260620 00:07:19.632802 9121 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:07:19.633776 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 9121
I20260620 00:07:19.634115 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38743
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37849
--webserver_port=37799
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:07:19.633286 9268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40735 every 8 connection(s)
Jun 20 00:07:19 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:19 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:19.646450 9271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:19.634129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.1:51159)
0620 00:07:19.634672 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:19.634725 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:19.636042 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:19.636699 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:19.636702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:19.637096 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:19.637393 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:19.637399 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.639322 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.639324 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:19.640556 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:19.640559 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.641618 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.641619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:19.641810 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:19.642582 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:19.642593 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:19.646134 (+ 3541us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":97,"threads_started":1}
W20260620 00:07:19.646503 9272 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:19.634165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51159 (local address 127.0.98.62:37849)
0620 00:07:19.634574 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:19.634586 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:19.634880 (+ 294us) server_negotiation.cc:407] Connection header received
0620 00:07:19.636318 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:19.636327 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:19.636393 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:19.636502 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:19.637544 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.639191 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.640683 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.641496 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.641638 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:19.646360 (+ 4722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51159: BlockingRecv error: recv got EOF from 127.0.98.1:51159 (error 108)
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":81,"threads_started":1}
W20260620 00:07:19.646822 9269 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37849 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:37849: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 20 00:07:19 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:19 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:19.653227 9271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:19.647133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.1:56009)
0620 00:07:19.647174 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:19.647186 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:19.647277 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:19.647428 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:19.647429 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:19.647610 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:19.647666 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:19.647668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.648589 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.648591 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:19.649415 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:19.649419 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.649856 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.649857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:19.649927 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:19.650470 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:19.650476 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:19.653132 (+ 2656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":19}
W20260620 00:07:19.653491 9272 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:19.647198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56009 (local address 127.0.98.62:37849)
0620 00:07:19.647283 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:19.647285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:19.647304 (+ 19us) server_negotiation.cc:407] Connection header received
0620 00:07:19.647334 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:19.647335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:19.647351 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:19.647426 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:19.647872 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.648484 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.649527 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.649758 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.649854 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:19.653237 (+ 3383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56009: BlockingRecv error: recv got EOF from 127.0.98.1:56009 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 00:07:19 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:19 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:19.659775 9271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:19.654328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.1:59115)
0620 00:07:19.654375 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:19.654389 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:19.654466 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:19.654656 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:19.654657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:19.654837 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:19.654883 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:19.654885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.655841 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.655843 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:19.656414 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:19.656417 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.656839 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.656840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:19.656898 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:19.657470 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:19.657475 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:19.659685 (+ 2210us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 00:07:19.659986 9269 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 00:07:19.660100 9272 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:19.654398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59115 (local address 127.0.98.62:37849)
0620 00:07:19.654481 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:19.654484 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:19.654497 (+ 13us) server_negotiation.cc:407] Connection header received
0620 00:07:19.654546 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:19.654547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:19.654560 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:19.654650 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:19.655212 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.655728 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.656523 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.656732 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.656843 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:19.660032 (+ 3189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59115: BlockingRecv error: recv got EOF from 127.0.98.1:59115 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 00:07:19.728380 9273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:19.728624 9273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:19.728652 9273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:19.730022 9273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:19.730073 9273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:19.730103 9273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:07:19.731698 9273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38743
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37799
--tserver_master_addrs=127.0.98.62:37849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:19.731974 9273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:19.732232 9273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:19.737864 9278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:19.737885 9281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:19.738029 9279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:19.738371 9273 server_base.cc:1034] running on GCE node
I20260620 00:07:19.738590 9273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:19.738891 9273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:19.740108 9273 hybrid_clock.cc:648] HybridClock initialized: now 1781914039740084 us; error 45 us; skew 500 ppm
Jun 20 00:07:19 dist-test-slave-grqt krb5kdc[8538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914039, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:19.743350 9273 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 00:07:19.743960 9273 webserver.cc:460] Webserver started at http://127.0.98.2:37799/ using document root <none> and password file <none>
I20260620 00:07:19.744267 9273 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:19.744366 9273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:19.746093 9273 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:07:19.746742 9288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:19.746978 9273 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:19.747100 9273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-1/wal
uuid: "cd2fa9e037c44ccfafef734569a0e5fb"
format_stamp: "Formatted at 2026-06-20 00:07:19 on dist-test-slave-grqt"
server_key: "f2d34848c5af683ef4a1abfb4bb05094"
server_key_iv: "f67ac052505fd98f1dec21540fea9dab"
server_key_version: "encryptionkey@0"
I20260620 00:07:19.747503 9273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:19.783115 9273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:19.785902 9273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:19.786106 9273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:19.786384 9273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:07:19.786775 9273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:07:19.786844 9273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:19.786893 9273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:07:19.786922 9273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:19.794528 9273 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38743
I20260620 00:07:19.794636 9401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38743 every 8 connection(s)
I20260620 00:07:19.795027 9273 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 00:07:19 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:19.802282 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 9273
Jun 20 00:07:19 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:19.802600 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41411
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37849
--webserver_port=34609
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:07:19.803473 9404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:19.795782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.2:50229)
0620 00:07:19.795965 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:19.796019 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:19.796568 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:19.796900 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:19.796904 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:19.797653 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:19.797907 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:19.797912 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.798855 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.798856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:19.799711 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:19.799713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.800120 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.800121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:19.800289 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:19.800826 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:19.800834 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:19.803315 (+ 2481us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":75,"threads_started":1}
W20260620 00:07:19.803539 9272 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:19.795883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50229 (local address 127.0.98.62:37849)
0620 00:07:19.795976 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:19.795979 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:19.796113 (+ 134us) server_negotiation.cc:407] Connection header received
0620 00:07:19.796691 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:19.796692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:19.796706 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:19.796769 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:19.798023 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.798744 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.799817 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.800012 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.800082 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:19.803447 (+ 3365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50229: BlockingRecv error: recv got EOF from 127.0.98.2:50229 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 00:07:19.803802 9402 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37849 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:37849: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 20 00:07:19 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:19 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:19.809199 9404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:19.804060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.2:50779)
0620 00:07:19.804122 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:19.804134 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:19.804186 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:19.804396 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:19.804397 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:19.804526 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:19.804572 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:19.804573 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.805221 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.805222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:19.806147 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:19.806149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.806524 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.806525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:19.806607 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:19.807106 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:19.807113 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:19.809108 (+ 1995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 00:07:19.809254 9272 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:19.804125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50779 (local address 127.0.98.62:37849)
0620 00:07:19.804182 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:19.804184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:19.804199 (+ 15us) server_negotiation.cc:407] Connection header received
0620 00:07:19.804296 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:19.804297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:19.804308 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:19.804390 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:19.804653 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.805136 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.806242 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.806435 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.806500 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:19.809187 (+ 2687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50779: BlockingRecv error: recv got EOF from 127.0.98.2:50779 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Jun 20 00:07:19 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:19 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:19.814314 9404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:19.809503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.2:35385)
0620 00:07:19.809684 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:19.809694 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:19.809742 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:19.809893 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:19.809894 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:19.810034 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:19.810070 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:19.810071 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.810739 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.810740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:19.811241 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:19.811243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.811615 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.811616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:19.811685 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:19.812200 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:19.812204 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:19.814236 (+ 2032us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":120,"threads_started":1}
W20260620 00:07:19.814457 9272 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:19.809540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35385 (local address 127.0.98.62:37849)
0620 00:07:19.809579 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:19.809581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:19.809729 (+ 148us) server_negotiation.cc:407] Connection header received
0620 00:07:19.809802 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:19.809803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:19.809813 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:19.809890 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:19.810153 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.810653 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.811334 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.811524 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.811615 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:19.814385 (+ 2770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35385: BlockingRecv error: recv got EOF from 127.0.98.2:35385 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 00:07:19.814479 9402 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 00:07:19.889736 9405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:19.889957 9405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:19.889984 9405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:19.891407 9405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:19.891533 9405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:19.891664 9405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:07:19.893532 9405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41411
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=34609
--tserver_master_addrs=127.0.98.62:37849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:19.893852 9405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:19.894140 9405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:19.899636 9411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:19.899637 9414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:19.899783 9405 server_base.cc:1034] running on GCE node
W20260620 00:07:19.899636 9412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:19.900203 9405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:19.900496 9405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:19.901672 9405 hybrid_clock.cc:648] HybridClock initialized: now 1781914039901656 us; error 27 us; skew 500 ppm
Jun 20 00:07:19 dist-test-slave-grqt krb5kdc[8538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914039, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:19.904269 9405 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 00:07:19.904799 9405 webserver.cc:460] Webserver started at http://127.0.98.3:34609/ using document root <none> and password file <none>
I20260620 00:07:19.905015 9405 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:19.905078 9405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:19.906746 9405 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:19.907455 9421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:19.907693 9405 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:19.907816 9405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-2/wal
uuid: "4de626faa66544378c59d7e6ad5d9553"
format_stamp: "Formatted at 2026-06-20 00:07:19 on dist-test-slave-grqt"
server_key: "b95afc73d7fe29160f3b059e8dc5b22a"
server_key_iv: "fc3afc0b7cf6abfd082cca41a37412d4"
server_key_version: "encryptionkey@0"
I20260620 00:07:19.908164 9405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:19.923734 9405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:19.926227 9405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:19.926355 9405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:19.926550 9405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:07:19.926821 9405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:07:19.926867 9405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:19.926889 9405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:07:19.926903 9405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:19.932934 9405 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41411
I20260620 00:07:19.933017 9534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41411 every 8 connection(s)
I20260620 00:07:19.933348 9405 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:07:19.938992 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 9405
Jun 20 00:07:19 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:19 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 00:07:19.941730 9272 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:19.934235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44299 (local address 127.0.98.62:37849)
0620 00:07:19.934322 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:19.934324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:19.934513 (+ 189us) server_negotiation.cc:407] Connection header received
0620 00:07:19.935039 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:19.935040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:19.935051 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:19.935106 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:19.936101 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.936610 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.937694 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.937895 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.937962 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:19.941669 (+ 3707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44299: BlockingRecv error: recv got EOF from 127.0.98.3:44299 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 00:07:19.941828 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:19.934210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.3:44299)
0620 00:07:19.934404 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:19.934442 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:19.934934 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:19.935163 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:19.935166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:19.935785 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:19.935996 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:19.936002 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.936704 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.936705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:19.937580 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:19.937583 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.938018 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.938019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:19.938139 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:19.938693 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:19.938701 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:19.941599 (+ 2898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":127,"threads_started":1}
W20260620 00:07:19.942020 9535 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37849 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:37849: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 20 00:07:19 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:19 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:19.947525 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:19.939585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34748 (local address 127.0.98.62:37849)
0620 00:07:19.939730 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:19.939732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:19.939741 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:07:19.939763 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:19.939764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:19.939775 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:19.939855 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:19.940635 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.941185 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.942180 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.942506 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.942579 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:19.943895 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:19.943900 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:19.943900 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:19.943914 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:19.945898 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:19.946352 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:19.946355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:19.946356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:19.946506 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:19.946745 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:19.946746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:19.946746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:19.947007 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:19.947178 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:19.947298 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 00:07:19.947422 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
I20260620 00:07:19.947527 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:19.942266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.3:36701)
0620 00:07:19.942326 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:19.942338 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:19.942374 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:19.942510 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:19.942511 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:19.942672 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:19.942720 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:19.942722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.943603 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.943603 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:19.944328 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:19.944331 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.944816 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.944817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:19.944893 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:19.945256 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:19.945261 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:19.947431 (+ 2170us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 00:07:19.947969 9272 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:19.942321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36701 (local address 127.0.98.62:37849)
0620 00:07:19.942355 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:19.942356 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:19.942365 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:07:19.942433 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:19.942434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:19.942441 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:19.942486 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:19.942806 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.943511 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.944502 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.944693 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.944814 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:19.947511 (+ 2697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36701: BlockingRecv error: recv got EOF from 127.0.98.3:36701 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
Jun 20 00:07:19 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:19 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:19.952782 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:19.948173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.3:46309)
0620 00:07:19.948236 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:19.948249 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:19.948294 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:19.948495 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:19.948496 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:19.948630 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:19.948664 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:19.948665 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.949341 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.949342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:19.949981 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:19.949983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.950347 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.950348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:19.950393 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:19.950820 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:19.950825 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:19.952695 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 00:07:19.952857 9272 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:19.948285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46309 (local address 127.0.98.62:37849)
0620 00:07:19.948394 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:19.948395 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:19.948401 (+ 6us) server_negotiation.cc:407] Connection header received
0620 00:07:19.948414 (+ 13us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:19.948415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:19.948422 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:19.948462 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:19.948738 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.949268 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.950057 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:19.950290 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:19.950348 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:19.952795 (+ 2447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46309: BlockingRecv error: recv got EOF from 127.0.98.3:46309 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260620 00:07:19.953001 9535 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 00:07:20 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:20 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:20.665759 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:20.660560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.1:33555)
0620 00:07:20.660742 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:20.660756 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:20.660840 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:20.661046 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:20.661047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:20.661321 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:20.661411 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:20.661413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:20.662158 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:20.662159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:20.662728 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:20.662730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:20.663125 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:20.663126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:20.663186 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:20.663743 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:20.663747 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:20.665647 (+ 1900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":100,"threads_started":1}
W20260620 00:07:20.665822 9541 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:20.660692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33555 (local address 127.0.98.62:37849)
0620 00:07:20.660889 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:20.660891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:20.660901 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:07:20.660931 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:20.660932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:20.660946 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:20.661035 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:20.661500 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:20.662073 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:20.662836 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:20.663038 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:20.663123 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:20.665745 (+ 2622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33555: BlockingRecv error: recv got EOF from 127.0.98.1:33555 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":76,"threads_started":1}
Jun 20 00:07:20 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:20 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:20.819901 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:20.815024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.2:55623)
0620 00:07:20.815210 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:20.815230 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:20.815308 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:20.815491 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:20.815492 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:20.815655 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:20.815702 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:20.815703 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:20.816355 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:20.816356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:20.816898 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:20.816899 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:20.817338 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:20.817339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:20.817443 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:20.817897 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:20.817901 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:20.819769 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":89,"threads_started":1}
W20260620 00:07:20.819928 9541 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:20.815161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55623 (local address 127.0.98.62:37849)
0620 00:07:20.815246 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:20.815248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:20.815351 (+ 103us) server_negotiation.cc:407] Connection header received
0620 00:07:20.815384 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:20.815385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:20.815397 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:20.815456 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:20.815796 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:20.816283 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:20.817004 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:20.817243 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:20.817334 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:20.819863 (+ 2529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55623: BlockingRecv error: recv got EOF from 127.0.98.2:55623 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 00:07:20 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:20 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:20.958529 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:20.953440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.3:54531)
0620 00:07:20.953591 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:20.953606 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:20.953669 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:20.953805 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:20.953806 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:20.953995 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:20.954042 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:20.954043 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:20.954727 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:20.954728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:20.955265 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:20.955267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:20.955676 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:20.955677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:20.955744 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:20.956229 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:20.956234 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:20.958392 (+ 2158us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":95,"threads_started":1}
W20260620 00:07:20.958599 9541 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:20.953576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54531 (local address 127.0.98.62:37849)
0620 00:07:20.953649 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:20.953651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:20.953662 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:07:20.953727 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:20.953728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:20.953741 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:20.953788 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:20.954131 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:20.954629 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:20.955385 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:20.955578 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:20.955673 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:20.958535 (+ 2862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54531: BlockingRecv error: recv got EOF from 127.0.98.3:54531 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 00:07:21 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:21 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:21.672384 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:21.666311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.1:38881)
0620 00:07:21.666483 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:21.666497 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:21.666583 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:21.666807 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:21.666809 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:21.667076 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:21.667156 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:21.667159 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:21.668094 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:21.668095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:21.668965 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:21.668968 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:21.669476 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:21.669477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:21.669538 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:21.670132 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:21.670137 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:21.672249 (+ 2112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":80,"threads_started":1}
W20260620 00:07:21.672453 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:21.666397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38881 (local address 127.0.98.62:37849)
0620 00:07:21.666516 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:21.666518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:21.666559 (+ 41us) server_negotiation.cc:407] Connection header received
0620 00:07:21.666676 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:21.666678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:21.666693 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:21.666760 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:21.667266 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:21.667997 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:21.669073 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:21.669343 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:21.669474 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:21.672353 (+ 2879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38881: BlockingRecv error: recv got EOF from 127.0.98.1:38881 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":69,"threads_started":1}
Jun 20 00:07:21 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:21 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:21.825892 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:21.820563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.2:58971)
0620 00:07:21.820853 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:21.820866 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:21.820977 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:21.821143 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:21.821144 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:21.821355 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:21.821473 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:21.821475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:21.822167 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:21.822168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:21.822716 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:21.822717 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:21.823108 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:21.823109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:21.823170 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:21.823671 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:21.823676 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:21.825825 (+ 2149us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":119,"threads_started":1}
W20260620 00:07:21.826061 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:21.820732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58971 (local address 127.0.98.62:37849)
0620 00:07:21.820853 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:21.820856 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:21.820957 (+ 101us) server_negotiation.cc:407] Connection header received
0620 00:07:21.821042 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:21.821043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:21.821058 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:21.821137 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:21.821559 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:21.822064 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:21.822817 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:21.823014 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:21.823107 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:21.825960 (+ 2853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58971: BlockingRecv error: recv got EOF from 127.0.98.2:58971 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Jun 20 00:07:21 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:21 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:21.963749 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:21.959071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.3:53451)
0620 00:07:21.959200 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:21.959219 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:21.959308 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:21.959469 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:21.959470 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:21.959648 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:21.959694 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:21.959696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:21.960408 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:21.960410 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:21.960960 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:21.960961 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:21.961330 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:21.961331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:21.961418 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:21.961922 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:21.961927 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:21.963633 (+ 1706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":76,"threads_started":1}
W20260620 00:07:21.963769 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:21.959227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53451 (local address 127.0.98.62:37849)
0620 00:07:21.959316 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:21.959318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:21.959331 (+ 13us) server_negotiation.cc:407] Connection header received
0620 00:07:21.959369 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:21.959370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:21.959388 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:21.959451 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:21.959789 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:21.960309 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:21.961062 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:21.961254 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:21.961332 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:21.963711 (+ 2379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53451: BlockingRecv error: recv got EOF from 127.0.98.3:53451 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 00:07:22 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:22 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:22.678052 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:22.672951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.1:46101)
0620 00:07:22.673118 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:22.673129 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:22.673189 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:22.673348 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:22.673349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:22.673546 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:22.673594 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:22.673596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:22.674389 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:22.674390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:22.674928 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:22.674930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:22.675454 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:22.675455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:22.675506 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:22.676099 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:22.676104 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:22.677983 (+ 1879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":103,"threads_started":1}
W20260620 00:07:22.678282 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:22.673044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46101 (local address 127.0.98.62:37849)
0620 00:07:22.673173 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:22.673175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:22.673197 (+ 22us) server_negotiation.cc:407] Connection header received
0620 00:07:22.673250 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:22.673250 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:22.673261 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:22.673345 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:22.673696 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:22.674248 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:22.675047 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:22.675327 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:22.675403 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:22.678208 (+ 2805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46101: BlockingRecv error: recv got EOF from 127.0.98.1:46101 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
Jun 20 00:07:22 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:22 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:22.831840 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:22.826512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.2:56433)
0620 00:07:22.826663 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:22.826675 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:22.826747 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:22.826927 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:22.826928 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:22.827089 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:22.827145 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:22.827147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:22.828074 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:22.828075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:22.828826 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:22.828829 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:22.829205 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:22.829206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:22.829244 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:22.829835 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:22.829843 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:22.831771 (+ 1928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
W20260620 00:07:22.832080 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:22.826628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56433 (local address 127.0.98.62:37849)
0620 00:07:22.826700 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:22.826702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:22.826734 (+ 32us) server_negotiation.cc:407] Connection header received
0620 00:07:22.826813 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:22.826813 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:22.826825 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:22.826880 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:22.827281 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:22.827980 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:22.828918 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:22.829130 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:22.829192 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:22.832016 (+ 2824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56433: BlockingRecv error: recv got EOF from 127.0.98.2:56433 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 00:07:22 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:22 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:22.969967 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:22.964340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.3:52351)
0620 00:07:22.964534 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:22.964545 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:22.964649 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:22.964822 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:22.964824 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:22.964983 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:22.965033 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:22.965034 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:22.965815 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:22.965816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:22.966388 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:22.966390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:22.966833 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:22.966834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:22.966890 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:22.967643 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:22.967648 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:22.969904 (+ 2256us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":58,"threads_started":1}
W20260620 00:07:22.970206 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:22.964460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52351 (local address 127.0.98.62:37849)
0620 00:07:22.964499 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:22.964501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:22.964652 (+ 151us) server_negotiation.cc:407] Connection header received
0620 00:07:22.964719 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:22.964720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:22.964731 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:22.964773 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:22.965106 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:22.965702 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:22.966491 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:22.966682 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:22.966798 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:22.970141 (+ 3343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52351: BlockingRecv error: recv got EOF from 127.0.98.3:52351 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
Jun 20 00:07:23 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:23 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:23.683084 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:23.678564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.1:56265)
0620 00:07:23.678709 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:23.678720 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:23.678784 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:23.678927 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:23.678928 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:23.679094 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:23.679158 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:23.679159 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:23.679782 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:23.679783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:23.680310 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:23.680312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:23.680629 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:23.680630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:23.680669 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:23.681114 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:23.681118 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:23.682987 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":79,"threads_started":1}
W20260620 00:07:23.683147 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:23.678664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56265 (local address 127.0.98.62:37849)
0620 00:07:23.678785 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:23.678787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:23.678822 (+ 35us) server_negotiation.cc:407] Connection header received
0620 00:07:23.678845 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:23.678845 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:23.678855 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:23.678919 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:23.679227 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:23.679715 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:23.680380 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:23.680566 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:23.680616 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:23.683076 (+ 2460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56265: BlockingRecv error: recv got EOF from 127.0.98.1:56265 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":69,"threads_started":1}
Jun 20 00:07:23 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:23 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:23.837409 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:23.832392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.2:41865)
0620 00:07:23.832588 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:23.832601 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:23.832699 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:23.832895 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:23.832896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:23.833078 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:23.833124 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:23.833126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:23.833861 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:23.833862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:23.834425 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:23.834427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:23.834833 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:23.834834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:23.834891 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:23.835413 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:23.835418 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:23.837300 (+ 1882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":95,"threads_started":1}
W20260620 00:07:23.837555 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:23.832524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41865 (local address 127.0.98.62:37849)
0620 00:07:23.832593 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:23.832596 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:23.832679 (+ 83us) server_negotiation.cc:407] Connection header received
0620 00:07:23.832772 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:23.832773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:23.832811 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:23.832873 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:23.833211 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:23.833766 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:23.834519 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:23.834718 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:23.834831 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:23.837482 (+ 2651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41865: BlockingRecv error: recv got EOF from 127.0.98.2:41865 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 00:07:23 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:23 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:23.975699 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:23.970536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.3:42193)
0620 00:07:23.970756 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:23.970769 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:23.970874 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:23.971044 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:23.971045 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:23.971231 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:23.971276 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:23.971278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:23.971972 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:23.971973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:23.972544 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:23.972546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:23.972929 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:23.972930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:23.972985 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:23.973560 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:23.973566 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:23.975638 (+ 2072us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":101,"threads_started":1}
W20260620 00:07:23.975838 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:23.970667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42193 (local address 127.0.98.62:37849)
0620 00:07:23.970743 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:23.970746 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:23.970854 (+ 108us) server_negotiation.cc:407] Connection header received
0620 00:07:23.970939 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:23.970940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:23.970955 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:23.971037 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:23.971365 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:23.971861 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:23.972640 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:23.972834 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:23.972928 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:23.975755 (+ 2827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42193: BlockingRecv error: recv got EOF from 127.0.98.3:42193 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 00:07:24 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:24 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:24.694774 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:24.683653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.1:35117)
0620 00:07:24.683793 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:24.683805 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:24.683867 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:24.684112 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:24.684113 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:24.684297 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:24.684345 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:24.684346 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:24.685117 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:24.685118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:24.685785 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:24.685788 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:24.686195 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:24.686196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:24.686249 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:24.692751 (+ 6502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:24.692756 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:24.694710 (+ 1954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":112,"threads_started":1}
W20260620 00:07:24.694970 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:24.683766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35117 (local address 127.0.98.62:37849)
0620 00:07:24.683916 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:24.683918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:24.683929 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:07:24.683961 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:24.683962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:24.683979 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:24.684054 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:24.684452 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:24.685016 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:24.685896 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:24.686098 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:24.686166 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:24.694867 (+ 8701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35117: BlockingRecv error: recv got EOF from 127.0.98.1:35117 (error 108)
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":113,"threads_started":1}
Jun 20 00:07:24 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:24 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:24.842767 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:24.837981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.2:60385)
0620 00:07:24.838180 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:24.838193 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:24.838259 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:24.838462 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:24.838464 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:24.838634 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:24.838682 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:24.838683 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:24.839369 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:24.839371 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:24.839907 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:24.839909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:24.840311 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:24.840312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:24.840374 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:24.840869 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:24.840873 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:24.842661 (+ 1788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":85,"threads_started":1}
W20260620 00:07:24.842821 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:24.838104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60385 (local address 127.0.98.62:37849)
0620 00:07:24.838153 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:24.838155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:24.838268 (+ 113us) server_negotiation.cc:407] Connection header received
0620 00:07:24.838323 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:24.838324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:24.838338 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:24.838418 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:24.838767 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:24.839273 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:24.840013 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:24.840208 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:24.840308 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:24.842755 (+ 2447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60385: BlockingRecv error: recv got EOF from 127.0.98.2:60385 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Jun 20 00:07:24 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:24 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:24.981668 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:24.976283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.3:52373)
0620 00:07:24.976473 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:24.976485 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:24.976586 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:24.976744 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:24.976745 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:24.976935 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:24.976981 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:24.976983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:24.977750 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:24.977752 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:24.978305 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:24.978307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:24.978706 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:24.978707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:24.978766 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:24.979301 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:24.979306 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:24.981580 (+ 2274us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":93,"threads_started":1}
W20260620 00:07:24.981822 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:24.976413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52373 (local address 127.0.98.62:37849)
0620 00:07:24.976472 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:24.976475 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:24.976569 (+ 94us) server_negotiation.cc:407] Connection header received
0620 00:07:24.976647 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:24.976648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:24.976662 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:24.976738 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:24.977072 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:24.977657 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:24.978399 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:24.978608 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:24.978701 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:24.981722 (+ 3021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52373: BlockingRecv error: recv got EOF from 127.0.98.3:52373 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Jun 20 00:07:25 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:25 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:25.700371 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:25.695357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.1:47351)
0620 00:07:25.695525 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:25.695536 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:25.695603 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:25.695761 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:25.695762 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:25.695938 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:25.695983 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:25.695985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:25.696738 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:25.696738 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:25.697271 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:25.697273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:25.697755 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:25.697756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:25.697798 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:25.698351 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:25.698355 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:25.700288 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":115,"threads_started":1}
W20260620 00:07:25.700557 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:25.695414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47351 (local address 127.0.98.62:37849)
0620 00:07:25.695566 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:25.695573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:25.695592 (+ 19us) server_negotiation.cc:407] Connection header received
0620 00:07:25.695663 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:25.695664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:25.695675 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:25.695749 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:25.696086 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:25.696657 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:25.697360 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:25.697683 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:25.697747 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:25.700458 (+ 2711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47351: BlockingRecv error: recv got EOF from 127.0.98.1:47351 (error 108)
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":93,"threads_started":1}
Jun 20 00:07:25 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:25 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:25.849541 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:25.843416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.2:49599)
0620 00:07:25.843602 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:25.843615 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:25.843679 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:25.843941 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:25.843943 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:25.844121 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:25.844170 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:25.844172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:25.844990 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:25.844991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:25.845695 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:25.845698 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:25.846175 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:25.846176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:25.846231 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:25.846804 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:25.846809 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:25.849434 (+ 2625us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":66,"threads_started":1}
W20260620 00:07:25.849694 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:25.843632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49599 (local address 127.0.98.62:37849)
0620 00:07:25.843726 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:25.843728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:25.843742 (+ 14us) server_negotiation.cc:407] Connection header received
0620 00:07:25.843805 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:25.843806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:25.843818 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:25.843883 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:25.844303 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:25.844859 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:25.845808 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:25.846057 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:25.846133 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:25.849585 (+ 3452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49599: BlockingRecv error: recv got EOF from 127.0.98.2:49599 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 00:07:25 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:25 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:25.987334 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:25.982337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.3:43281)
0620 00:07:25.982460 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:25.982504 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:25.982576 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:25.982709 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:25.982710 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:25.982866 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:25.982917 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:25.982919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:25.983585 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:25.983586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:25.984120 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:25.984122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:25.984465 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:25.984466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:25.984552 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:25.985085 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:25.985090 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:25.987226 (+ 2136us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":68,"threads_started":1}
W20260620 00:07:25.987423 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:25.982427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43281 (local address 127.0.98.62:37849)
0620 00:07:25.982484 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:25.982486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:25.982580 (+ 94us) server_negotiation.cc:407] Connection header received
0620 00:07:25.982623 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:25.982624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:25.982639 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:25.982689 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:25.982994 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:25.983498 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:25.984201 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:25.984384 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:25.984448 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:25.987341 (+ 2893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43281: BlockingRecv error: recv got EOF from 127.0.98.3:43281 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Jun 20 00:07:26 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:26 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:26.706007 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:26.700880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.1:52331)
0620 00:07:26.701034 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:26.701044 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:26.701097 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:26.701223 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:26.701224 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:26.701535 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:26.701585 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:26.701588 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:26.702379 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:26.702380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:26.702942 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:26.702943 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:26.703284 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:26.703285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:26.703325 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:26.703823 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:26.703835 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:26.705945 (+ 2110us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":81,"threads_started":1}
W20260620 00:07:26.706147 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:26.700948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52331 (local address 127.0.98.62:37849)
0620 00:07:26.701090 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:26.701092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:26.701103 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:07:26.701135 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:26.701135 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:26.701147 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:26.701212 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:26.701762 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:26.702279 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:26.703021 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:26.703211 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:26.703264 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:26.706055 (+ 2791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52331: BlockingRecv error: recv got EOF from 127.0.98.1:52331 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":92,"threads_started":1}
Jun 20 00:07:26 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:26 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:26.856205 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:26.850289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.2:45121)
0620 00:07:26.850455 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:26.850467 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:26.850543 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:26.850698 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:26.850699 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:26.850911 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:26.850959 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:26.850961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:26.851801 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:26.851802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:26.852382 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:26.852384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:26.853001 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:26.853002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:26.853062 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:26.853693 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:26.853698 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:26.856137 (+ 2439us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":109,"threads_started":1}
W20260620 00:07:26.856340 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:26.850391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45121 (local address 127.0.98.62:37849)
0620 00:07:26.850476 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:26.850478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:26.850531 (+ 53us) server_negotiation.cc:407] Connection header received
0620 00:07:26.850618 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:26.850619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:26.850631 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:26.850679 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:26.851082 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:26.851682 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:26.852682 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:26.852901 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:26.852976 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:26.856256 (+ 3280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45121: BlockingRecv error: recv got EOF from 127.0.98.2:45121 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 00:07:26 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:26 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:26.993432 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:26.987881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.3:53507)
0620 00:07:26.988139 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:26.988151 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:26.988232 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:26.988573 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:26.988574 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:26.988749 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:26.988800 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:26.988801 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:26.989703 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:26.989704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:26.990244 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:26.990246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:26.990712 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:26.990712 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:26.990792 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:26.991361 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:26.991366 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:26.993304 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":124,"threads_started":1}
W20260620 00:07:26.993688 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:26.988019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53507 (local address 127.0.98.62:37849)
0620 00:07:26.988075 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:26.988077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:26.988232 (+ 155us) server_negotiation.cc:407] Connection header received
0620 00:07:26.988300 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:26.988301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:26.988313 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:26.988379 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:26.988920 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:26.989557 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:26.990401 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:26.990610 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:26.990709 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:26.993507 (+ 2798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53507: BlockingRecv error: recv got EOF from 127.0.98.3:53507 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Jun 20 00:07:27 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:27 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:27.712198 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:27.706604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.1:45403)
0620 00:07:27.706766 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:27.706777 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:27.706828 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:27.707068 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:27.707069 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:27.707230 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:27.707315 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:27.707317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:27.708067 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:27.708068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:27.708632 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:27.708634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:27.709079 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:27.709080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:27.709135 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:27.709693 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:27.709697 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:27.712097 (+ 2400us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":97,"threads_started":1}
W20260620 00:07:27.712265 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:27.706752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45403 (local address 127.0.98.62:37849)
0620 00:07:27.706905 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:27.706907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:27.706915 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:07:27.706940 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:27.706941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:27.706951 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:27.707022 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:27.707444 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:27.707973 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:27.708742 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:27.708952 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:27.709020 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:27.712193 (+ 3173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45403: BlockingRecv error: recv got EOF from 127.0.98.1:45403 (error 108)
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":55,"threads_started":1}
Jun 20 00:07:27 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:27 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:27.863588 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:27.856947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.2:55433)
0620 00:07:27.857249 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:27.857264 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:27.857360 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:27.857966 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:27.857967 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:27.858312 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:27.858376 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:27.858380 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:27.859513 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:27.859514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:27.860088 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:27.860090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:27.860673 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:27.860675 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:27.860733 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:27.861276 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:27.861281 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:27.863501 (+ 2220us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":111,"threads_started":1}
W20260620 00:07:27.863775 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:27.857093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55433 (local address 127.0.98.62:37849)
0620 00:07:27.857175 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:27.857177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:27.857402 (+ 225us) server_negotiation.cc:407] Connection header received
0620 00:07:27.857783 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:27.857785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:27.857801 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:27.857879 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:27.858501 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:27.859394 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:27.860359 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:27.860566 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:27.860640 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:27.863656 (+ 3016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55433: BlockingRecv error: recv got EOF from 127.0.98.2:55433 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 00:07:27 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:27 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:28.000061 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:27.994089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.3:33877)
0620 00:07:27.994240 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:27.994251 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:27.994318 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:27.994602 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:27.994604 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:27.994773 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:27.994823 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:27.994825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:27.995640 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:27.995641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:27.996279 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:27.996282 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:27.996978 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:27.996979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:27.997033 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:27.997754 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:27.997760 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:27.999996 (+ 2236us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":78,"threads_started":1}
W20260620 00:07:28.000295 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:27.994173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33877 (local address 127.0.98.62:37849)
0620 00:07:27.994250 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:27.994252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:27.994317 (+ 65us) server_negotiation.cc:407] Connection header received
0620 00:07:27.994394 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:27.994395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:27.994416 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:27.994594 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:27.994953 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:27.995536 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:27.996669 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:27.996873 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:27.996938 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:28.000217 (+ 3279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33877: BlockingRecv error: recv got EOF from 127.0.98.3:33877 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 00:07:28 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:28 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:28.717931 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:28.712855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.1:41949)
0620 00:07:28.713025 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:28.713037 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:28.713098 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:28.713259 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:28.713259 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:28.713496 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:28.713546 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:28.713547 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:28.714585 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:28.714586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:28.715125 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:28.715127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:28.715480 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:28.715482 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:28.715528 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:28.716034 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:28.716046 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:28.717873 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":99,"threads_started":1}
W20260620 00:07:28.718102 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:28.712986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41949 (local address 127.0.98.62:37849)
0620 00:07:28.713117 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:28.713119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:28.713130 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:07:28.713167 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:28.713168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:28.713179 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:28.713243 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:28.713657 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:28.714512 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:28.715198 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:28.715415 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:28.715472 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:28.718000 (+ 2528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41949: BlockingRecv error: recv got EOF from 127.0.98.1:41949 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
Jun 20 00:07:28 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:28 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:28.869239 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:28.864233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.2:48845)
0620 00:07:28.864391 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:28.864401 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:28.864449 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:28.864608 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:28.864609 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:28.864771 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:28.864847 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:28.864848 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:28.865643 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:28.865644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:28.866175 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:28.866177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:28.866600 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:28.866601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:28.866653 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:28.867199 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:28.867203 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:28.869179 (+ 1976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":63,"threads_started":1}
W20260620 00:07:28.869408 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:28.864379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48845 (local address 127.0.98.62:37849)
0620 00:07:28.864427 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:28.864429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:28.864441 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:07:28.864505 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:28.864505 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:28.864516 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:28.864580 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:28.864958 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:28.865514 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:28.866271 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:28.866502 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:28.866563 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:28.869291 (+ 2728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48845: BlockingRecv error: recv got EOF from 127.0.98.2:48845 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Jun 20 00:07:28 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:28 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:29.005728 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:29.000856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.3:56643)
0620 00:07:29.001029 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:29.001042 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:29.001133 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:29.001303 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:29.001304 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:29.001551 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:29.001607 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:29.001609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:29.002277 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:29.002278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:29.002798 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:29.002799 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:29.003167 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:29.003168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:29.003224 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:29.003712 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:29.003717 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:29.005628 (+ 1911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":105,"threads_started":1}
W20260620 00:07:29.005770 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:29.000980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56643 (local address 127.0.98.62:37849)
0620 00:07:29.001034 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:29.001036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:29.001113 (+ 77us) server_negotiation.cc:407] Connection header received
0620 00:07:29.001207 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:29.001208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:29.001223 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:29.001297 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:29.001698 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:29.002188 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:29.002894 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:29.003086 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:29.003167 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:29.005706 (+ 2539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56643: BlockingRecv error: recv got EOF from 127.0.98.3:56643 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Jun 20 00:07:29 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:29 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:29.728488 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:29.718580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.1:43607)
0620 00:07:29.718750 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:29.718761 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:29.718840 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:29.719050 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:29.719051 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:29.719218 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:29.719265 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:29.719266 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:29.720053 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:29.720054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:29.720614 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:29.720616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:29.721011 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:29.721012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:29.721053 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:29.726307 (+ 5254us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:29.726313 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:29.728422 (+ 2109us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":84,"threads_started":1}
W20260620 00:07:29.728684 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:29.718628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43607 (local address 127.0.98.62:37849)
0620 00:07:29.718753 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:29.718756 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:29.718868 (+ 112us) server_negotiation.cc:407] Connection header received
0620 00:07:29.718904 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:29.718905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:29.718918 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:29.718989 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:29.719379 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:29.719953 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:29.720721 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:29.720937 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:29.721002 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:29.728578 (+ 7576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43607: BlockingRecv error: recv got EOF from 127.0.98.1:43607 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
Jun 20 00:07:29 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:29 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:29.875145 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:29.869742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.2:35543)
0620 00:07:29.870059 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:29.870072 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:29.870172 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:29.870335 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:29.870337 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:29.870727 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:29.870783 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:29.870785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:29.871515 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:29.871516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:29.872053 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:29.872055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:29.872442 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:29.872443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:29.872498 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:29.873015 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:29.873019 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:29.875086 (+ 2067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":86,"threads_started":1}
W20260620 00:07:29.875308 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:29.869847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35543 (local address 127.0.98.62:37849)
0620 00:07:29.870037 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:29.870040 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:29.870151 (+ 111us) server_negotiation.cc:407] Connection header received
0620 00:07:29.870236 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:29.870237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:29.870251 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:29.870330 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:29.870919 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:29.871426 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:29.872156 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:29.872357 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:29.872442 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:29.875209 (+ 2767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35543: BlockingRecv error: recv got EOF from 127.0.98.2:35543 (error 108)
Metrics: {"server-negotiator.queue_time_us":158}
Jun 20 00:07:30 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:30 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:30.011472 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:30.006331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.3:34071)
0620 00:07:30.006501 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:30.006519 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:30.006576 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:30.006768 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:30.006770 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:30.006954 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:30.007001 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:30.007004 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:30.007680 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:30.007681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:30.008202 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:30.008205 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:30.008579 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:30.008580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:30.008678 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:30.009191 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:30.009196 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:30.011372 (+ 2176us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":60,"threads_started":1}
W20260620 00:07:30.011539 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:30.006562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34071 (local address 127.0.98.62:37849)
0620 00:07:30.006632 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:30.006634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:30.006643 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:07:30.006678 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:30.006679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:30.006689 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:30.006758 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:30.007097 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:30.007594 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:30.008305 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:30.008502 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:30.008580 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:30.011473 (+ 2893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34071: BlockingRecv error: recv got EOF from 127.0.98.3:34071 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 00:07:30 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:30 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:30.734357 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:30.729186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.1:53257)
0620 00:07:30.729335 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:30.729352 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:30.729481 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:30.729680 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:30.729682 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:30.729851 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:30.729899 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:30.729902 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:30.730619 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:30.730620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:30.731167 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:30.731169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:30.731612 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:30.731613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:30.731663 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:30.732236 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:30.732240 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:30.734278 (+ 2038us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":96,"threads_started":1}
W20260620 00:07:30.734488 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:30.729254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53257 (local address 127.0.98.62:37849)
0620 00:07:30.729429 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:30.729463 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:30.729475 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:07:30.729547 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:30.729547 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:30.729559 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:30.729632 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:30.730002 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:30.730537 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:30.731274 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:30.731512 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:30.731577 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:30.734359 (+ 2782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53257: BlockingRecv error: recv got EOF from 127.0.98.1:53257 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":86,"threads_started":1}
Jun 20 00:07:30 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:30 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:30.880746 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:30.875760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.2:41115)
0620 00:07:30.875923 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:30.875933 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:30.875980 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:30.876159 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:30.876160 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:30.876331 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:30.876379 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:30.876380 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:30.877040 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:30.877040 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:30.877660 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:30.877663 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:30.878016 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:30.878017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:30.878055 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:30.878543 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:30.878548 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:30.880686 (+ 2138us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":68,"threads_started":1}
W20260620 00:07:30.880959 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:30.875903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41115 (local address 127.0.98.62:37849)
0620 00:07:30.875978 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:30.875980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:30.875990 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:07:30.876024 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:30.876025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:30.876035 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:30.876101 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:30.876460 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:30.876967 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:30.877743 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:30.877931 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:30.878001 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:30.880891 (+ 2890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41115: BlockingRecv error: recv got EOF from 127.0.98.2:41115 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Jun 20 00:07:31 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:31 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:31.017130 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:31.012070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.3:41193)
0620 00:07:31.012212 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:31.012224 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:31.012286 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:31.012516 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:31.012517 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:31.012704 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:31.012751 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:31.012753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:31.013502 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:31.013503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:31.014037 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:31.014038 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:31.014424 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:31.014425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:31.014488 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:31.014957 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:31.014962 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:31.017062 (+ 2100us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":93,"threads_started":1}
W20260620 00:07:31.017282 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:31.012270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41193 (local address 127.0.98.62:37849)
0620 00:07:31.012377 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:31.012379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:31.012388 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:07:31.012417 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:31.012417 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:31.012431 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:31.012510 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:31.012868 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:31.013433 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:31.014123 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:31.014341 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:31.014414 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:31.017196 (+ 2782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41193: BlockingRecv error: recv got EOF from 127.0.98.3:41193 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 00:07:31 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:31 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:31.740959 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:31.734932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.1:47365)
0620 00:07:31.735099 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:31.735112 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:31.735169 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:31.735335 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:31.735336 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:31.735497 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:31.735562 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:31.735564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:31.736678 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:31.736679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:31.737253 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:31.737255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:31.737864 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:31.737865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:31.737927 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:31.738539 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:31.738544 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:31.740825 (+ 2281us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":79,"threads_started":1}
W20260620 00:07:31.740985 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:31.735030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47365 (local address 127.0.98.62:37849)
0620 00:07:31.735152 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:31.735154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:31.735164 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:07:31.735233 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:31.735234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:31.735249 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:31.735325 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:31.735638 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:31.736561 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:31.737341 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:31.737747 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:31.737831 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:31.740931 (+ 3100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47365: BlockingRecv error: recv got EOF from 127.0.98.1:47365 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":53,"threads_started":1}
Jun 20 00:07:31 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:31 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:31.886111 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:31.881300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.2:51899)
0620 00:07:31.881517 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:31.881539 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:31.881601 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:31.881769 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:31.881771 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:31.881964 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:31.882013 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:31.882015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:31.882683 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:31.882684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:31.883227 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:31.883228 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:31.883636 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:31.883637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:31.883686 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:31.884119 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:31.884124 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:31.886038 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":169,"threads_started":1}
W20260620 00:07:31.886247 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:31.881438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51899 (local address 127.0.98.62:37849)
0620 00:07:31.881487 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:31.881488 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:31.881585 (+ 97us) server_negotiation.cc:407] Connection header received
0620 00:07:31.881665 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:31.881666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:31.881677 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:31.881720 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:31.882093 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:31.882605 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:31.883322 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:31.883540 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:31.883611 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:31.886156 (+ 2545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51899: BlockingRecv error: recv got EOF from 127.0.98.2:51899 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Jun 20 00:07:32 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:32 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:32.023021 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:32.017754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.3:51711)
0620 00:07:32.017957 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:32.017968 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:32.018058 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:32.018226 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:32.018228 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:32.018456 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:32.018502 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:32.018504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:32.019186 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:32.019187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:32.019716 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:32.019718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:32.020098 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:32.020099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:32.020158 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:32.020711 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:32.020717 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:32.022948 (+ 2231us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":111,"threads_started":1}
W20260620 00:07:32.023149 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:32.017825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51711 (local address 127.0.98.62:37849)
0620 00:07:32.017930 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:32.017933 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:32.018044 (+ 111us) server_negotiation.cc:407] Connection header received
0620 00:07:32.018131 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:32.018133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:32.018153 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:32.018220 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:32.018594 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:32.019099 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:32.019808 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:32.020008 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:32.020095 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:32.023061 (+ 2966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51711: BlockingRecv error: recv got EOF from 127.0.98.3:51711 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 00:07:32 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:32 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:32.746301 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:32.741569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.1:39405)
0620 00:07:32.741728 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:32.741739 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:32.741814 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:32.742028 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:32.742030 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:32.742237 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:32.742287 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:32.742289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:32.743011 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:32.743012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:32.743600 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:32.743602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:32.743947 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:32.743948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:32.743994 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:32.744495 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:32.744499 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:32.746241 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":72,"threads_started":1}
W20260620 00:07:32.746528 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:32.741618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39405 (local address 127.0.98.62:37849)
0620 00:07:32.741733 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:32.741735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:32.741818 (+ 83us) server_negotiation.cc:407] Connection header received
0620 00:07:32.741912 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:32.741913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:32.741927 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:32.742004 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:32.742401 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:32.742924 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:32.743678 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:32.743877 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:32.743929 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:32.746425 (+ 2496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39405: BlockingRecv error: recv got EOF from 127.0.98.1:39405 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":54,"threads_started":1}
Jun 20 00:07:32 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:32 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:32.891736 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:32.886760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.2:36311)
0620 00:07:32.886931 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:32.886941 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:32.887025 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:32.887194 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:32.887196 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:32.887371 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:32.887419 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:32.887421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:32.888226 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:32.888227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:32.888759 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:32.888760 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:32.889230 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:32.889231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:32.889296 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:32.889764 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:32.889769 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:32.891669 (+ 1900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":100,"threads_started":1}
W20260620 00:07:32.891870 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:32.886933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36311 (local address 127.0.98.62:37849)
0620 00:07:32.887008 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:32.887010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:32.887022 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:07:32.887100 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:32.887101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:32.887120 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:32.887187 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:32.887502 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:32.888136 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:32.888848 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:32.889146 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:32.889220 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:32.891783 (+ 2563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36311: BlockingRecv error: recv got EOF from 127.0.98.2:36311 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 00:07:33 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:33 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:33.028486 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:33.023601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.3:46609)
0620 00:07:33.023714 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:33.023725 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:33.023790 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:33.023922 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:33.023923 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:33.024078 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:33.024127 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:33.024128 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:33.024777 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:33.024778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:33.025345 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:33.025346 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:33.025852 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:33.025853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:33.025906 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:33.026400 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:33.026405 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:33.028422 (+ 2017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":65,"threads_started":1}
W20260620 00:07:33.028673 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:33.023666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46609 (local address 127.0.98.62:37849)
0620 00:07:33.023739 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:33.023741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:33.023791 (+ 50us) server_negotiation.cc:407] Connection header received
0620 00:07:33.023834 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:33.023835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:33.023847 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:33.023902 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:33.024190 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:33.024682 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:33.025527 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:33.025750 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:33.025823 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:33.028589 (+ 2766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46609: BlockingRecv error: recv got EOF from 127.0.98.3:46609 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 00:07:33 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:33 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:33.752502 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:33.746944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.1:35357)
0620 00:07:33.747155 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:33.747166 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:33.747230 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:33.747384 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:33.747385 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:33.747548 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:33.747595 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:33.747596 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:33.748540 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:33.748541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:33.749411 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:33.749414 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:33.749765 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:33.749766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:33.749806 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:33.750418 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:33.750424 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:33.752440 (+ 2016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":154,"threads_started":1}
W20260620 00:07:33.752691 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:33.747103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35357 (local address 127.0.98.62:37849)
0620 00:07:33.747217 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:33.747219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:33.747230 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:07:33.747297 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:33.747298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:33.747309 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:33.747369 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:33.747678 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:33.748366 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:33.749490 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:33.749690 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:33.749750 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:33.752572 (+ 2822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35357: BlockingRecv error: recv got EOF from 127.0.98.1:35357 (error 108)
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":52,"threads_started":1}
Jun 20 00:07:33 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:33 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:33.897533 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:33.892353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.2:35897)
0620 00:07:33.892504 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:33.892518 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:33.892601 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:33.892826 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:33.892827 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:33.893096 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:33.893159 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:33.893161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:33.893854 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:33.893855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:33.894386 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:33.894388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:33.894803 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:33.894804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:33.894857 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:33.895336 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:33.895341 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:33.897465 (+ 2124us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":92,"threads_started":1}
W20260620 00:07:33.897581 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:33.892458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35897 (local address 127.0.98.62:37849)
0620 00:07:33.892511 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:33.892514 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:33.892578 (+ 64us) server_negotiation.cc:407] Connection header received
0620 00:07:33.892719 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:33.892719 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:33.892734 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:33.892798 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:33.893232 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:33.893769 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:33.894516 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:33.894705 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:33.894772 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:33.897533 (+ 2761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35897: BlockingRecv error: recv got EOF from 127.0.98.2:35897 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Jun 20 00:07:34 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:34 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:34.033891 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:34.029059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.3:34531)
0620 00:07:34.029229 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:34.029244 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:34.029333 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:34.029515 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:34.029516 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:34.029693 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:34.029738 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:34.029740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:34.030422 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:34.030424 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:34.030968 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:34.030969 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:34.031355 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:34.031356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:34.031411 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:34.031961 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:34.031966 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:34.033770 (+ 1804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
W20260620 00:07:34.033927 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:34.029173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34531 (local address 127.0.98.62:37849)
0620 00:07:34.029257 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:34.029259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:34.029327 (+ 68us) server_negotiation.cc:407] Connection header received
0620 00:07:34.029438 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:34.029439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:34.029451 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:34.029499 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:34.029824 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:34.030321 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:34.031061 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:34.031266 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:34.031353 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:34.033860 (+ 2507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34531: BlockingRecv error: recv got EOF from 127.0.98.3:34531 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 00:07:34 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:34 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:34.763621 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:34.753171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.1:40203)
0620 00:07:34.753336 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:34.753347 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:34.753467 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:34.753699 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:34.753700 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:34.753939 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:34.753994 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:34.753996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:34.754721 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:34.754722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:34.755288 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:34.755290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:34.755674 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:34.755675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:34.755733 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:34.761483 (+ 5750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:34.761489 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:34.763538 (+ 2049us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":56,"threads_started":1}
W20260620 00:07:34.763756 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:34.753345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40203 (local address 127.0.98.62:37849)
0620 00:07:34.753514 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:34.753517 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:34.753528 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:07:34.753559 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:34.753559 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:34.753572 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:34.753671 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:34.754080 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:34.754633 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:34.755377 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:34.755588 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:34.755673 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:34.763677 (+ 8004us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40203: BlockingRecv error: recv got EOF from 127.0.98.1:40203 (error 108)
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":118,"threads_started":1}
Jun 20 00:07:34 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:07:34 dist-test-slave-grqt krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:07:34.903316 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:34.898182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37849 (local address 127.0.98.2:43099)
0620 00:07:34.898349 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:34.898360 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:34.898455 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:34.898680 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:34.898681 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:34.898947 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:34.899008 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:34.899011 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:34.899769 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:34.899770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:34.900304 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:34.900306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:34.900707 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:34.900708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:34.900767 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:34.901270 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:34.901274 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:34.903235 (+ 1961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37849: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":96,"threads_started":1}
W20260620 00:07:34.903502 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:34.898324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43099 (local address 127.0.98.62:37849)
0620 00:07:34.898379 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:34.898381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:34.898433 (+ 52us) server_negotiation.cc:407] Connection header received
0620 00:07:34.898534 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:34.898535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:34.898548 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:34.898620 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:34.899112 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:34.899680 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:34.900417 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:34.900618 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:34.900706 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:34.903417 (+ 2711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43099: BlockingRecv error: recv got EOF from 127.0.98.2:43099 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 00:07:34.942878 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 9121
I20260620 00:07:34.947569 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 9273
I20260620 00:07:34.951978 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 9405
I20260620 00:07:34.956429 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 9051
2026-06-20T00:07:34Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17648 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T00:07:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:07:34Z Disabled control of system clock
I20260620 00:07:34.984416 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36181
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36181
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 00:07:35.072372 9602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:35.072623 9602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:35.072646 9602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:35.073941 9602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:07:35.073987 9602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:35.074000 9602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:35.074016 9602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:07:35.074030 9602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:07:35.075479 9602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36181
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:35.075688 9602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:35.075954 9602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:35.081311 9610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:35.081315 9608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:35.081315 9607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:35.081552 9602 server_base.cc:1034] running on GCE node
I20260620 00:07:35.081950 9602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:35.082223 9602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:35.083441 9602 hybrid_clock.cc:648] HybridClock initialized: now 1781914055083416 us; error 28 us; skew 500 ppm
I20260620 00:07:35.085124 9602 webserver.cc:460] Webserver started at http://127.0.98.62:38655/ using document root <none> and password file <none>
I20260620 00:07:35.085444 9602 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:35.085503 9602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:35.085659 9602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:35.086783 9602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "2900a1e76eb04fd9861b5b003511f43a"
format_stamp: "Formatted at 2026-06-20 00:07:35 on dist-test-slave-grqt"
server_key: "3e65553b2320c5ab7b998cffb8bf30c4"
server_key_iv: "caff087b2b7afb02c401c326eb38c683"
server_key_version: "encryptionkey@0"
I20260620 00:07:35.087170 9602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "2900a1e76eb04fd9861b5b003511f43a"
format_stamp: "Formatted at 2026-06-20 00:07:35 on dist-test-slave-grqt"
server_key: "3e65553b2320c5ab7b998cffb8bf30c4"
server_key_iv: "caff087b2b7afb02c401c326eb38c683"
server_key_version: "encryptionkey@0"
I20260620 00:07:35.088670 9602 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:07:35.089600 9616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:35.089833 9602 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:07:35.089962 9602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "2900a1e76eb04fd9861b5b003511f43a"
format_stamp: "Formatted at 2026-06-20 00:07:35 on dist-test-slave-grqt"
server_key: "3e65553b2320c5ab7b998cffb8bf30c4"
server_key_iv: "caff087b2b7afb02c401c326eb38c683"
server_key_version: "encryptionkey@0"
I20260620 00:07:35.090049 9602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:35.110400 9602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:35.110759 9602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:35.110934 9602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:35.114998 9602 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36181
I20260620 00:07:35.115032 9668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36181 every 8 connection(s)
I20260620 00:07:35.115396 9602 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:07:35.116115 9669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:35.118856 9669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2900a1e76eb04fd9861b5b003511f43a: Bootstrap starting.
I20260620 00:07:35.119357 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 9602
I20260620 00:07:35.119486 9669 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2900a1e76eb04fd9861b5b003511f43a: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:35.119568 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:07:35.119782 9669 log.cc:824] T 00000000000000000000000000000000 P 2900a1e76eb04fd9861b5b003511f43a: Log is configured to *not* fsync() on all Append() calls
I20260620 00:07:35.119807 392 external_mini_cluster.cc:1428] Setting key 144f7f11090aef8151b3a6d592951aee
I20260620 00:07:35.120689 9669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2900a1e76eb04fd9861b5b003511f43a: No bootstrap required, opened a new log
I20260620 00:07:35.123714 9669 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2900a1e76eb04fd9861b5b003511f43a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2900a1e76eb04fd9861b5b003511f43a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36181 } }
I20260620 00:07:35.123883 9669 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2900a1e76eb04fd9861b5b003511f43a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:35.123910 9669 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2900a1e76eb04fd9861b5b003511f43a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2900a1e76eb04fd9861b5b003511f43a, State: Initialized, Role: FOLLOWER
I20260620 00:07:35.124068 9669 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2900a1e76eb04fd9861b5b003511f43a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2900a1e76eb04fd9861b5b003511f43a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36181 } }
I20260620 00:07:35.124167 9669 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2900a1e76eb04fd9861b5b003511f43a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:07:35.124204 9669 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2900a1e76eb04fd9861b5b003511f43a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:07:35.124291 9669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2900a1e76eb04fd9861b5b003511f43a [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:35.124986 9669 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2900a1e76eb04fd9861b5b003511f43a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2900a1e76eb04fd9861b5b003511f43a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36181 } }
I20260620 00:07:35.125120 9669 leader_election.cc:302] T 00000000000000000000000000000000 P 2900a1e76eb04fd9861b5b003511f43a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2900a1e76eb04fd9861b5b003511f43a; no voters:
I20260620 00:07:35.125339 9669 leader_election.cc:288] T 00000000000000000000000000000000 P 2900a1e76eb04fd9861b5b003511f43a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:07:35.125496 9674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2900a1e76eb04fd9861b5b003511f43a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:07:35.125670 9674 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2900a1e76eb04fd9861b5b003511f43a [term 1 LEADER]: Becoming Leader. State: Replica: 2900a1e76eb04fd9861b5b003511f43a, State: Running, Role: LEADER
I20260620 00:07:35.125756 9669 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2900a1e76eb04fd9861b5b003511f43a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:07:35.125867 9674 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2900a1e76eb04fd9861b5b003511f43a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2900a1e76eb04fd9861b5b003511f43a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36181 } }
I20260620 00:07:35.126291 9675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2900a1e76eb04fd9861b5b003511f43a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2900a1e76eb04fd9861b5b003511f43a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2900a1e76eb04fd9861b5b003511f43a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36181 } } }
I20260620 00:07:35.126585 9675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2900a1e76eb04fd9861b5b003511f43a [sys.catalog]: This master's current role is: LEADER
I20260620 00:07:35.126494 9676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2900a1e76eb04fd9861b5b003511f43a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2900a1e76eb04fd9861b5b003511f43a. Latest consensus state: current_term: 1 leader_uuid: "2900a1e76eb04fd9861b5b003511f43a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2900a1e76eb04fd9861b5b003511f43a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36181 } } }
I20260620 00:07:35.126809 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:35.120385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56738 (local address 127.0.98.62:36181)
0620 00:07:35.120620 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:35.120630 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:35.120667 (+ 37us) server_negotiation.cc:407] Connection header received
0620 00:07:35.121272 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:35.121283 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:35.121375 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:35.121573 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:35.121739 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:35.123301 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:35.124273 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:35.124798 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:35.124981 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:35.125772 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:35.125776 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:35.125776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:07:35.126026 (+ 250us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:35.126076 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:07:35.126092 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:35.126201 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:35.126477 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 00:07:35.126566 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":90,"threads_started":1}
I20260620 00:07:35.126905 9676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2900a1e76eb04fd9861b5b003511f43a [sys.catalog]: This master's current role is: LEADER
I20260620 00:07:35.127089 9683 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:07:35.127650 9683 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:07:35.129317 9683 catalog_manager.cc:1347] Generated new cluster ID: eeacc689e1b843ccb2d2029a2ecd7ae1
I20260620 00:07:35.129420 9683 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:07:35.133003 9683 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:07:35.133616 9683 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:07:35.135699 9683 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2900a1e76eb04fd9861b5b003511f43a: Generated new TSK 0
I20260620 00:07:35.135882 9683 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:07:35.142362 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36181
--builtin_ntp_servers=127.0.98.20:43403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 00:07:35.251492 9693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:35.251763 9693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:35.251799 9693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:35.253034 9693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:35.253104 9693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:35.253147 9693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:07:35.254640 9693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:35.254932 9693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:35.255244 9693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:35.261538 9699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:35.261529 9698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:35.261598 9693 server_base.cc:1034] running on GCE node
W20260620 00:07:35.261529 9701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:35.261880 9693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:35.262212 9693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:35.263418 9693 hybrid_clock.cc:648] HybridClock initialized: now 1781914055263381 us; error 65 us; skew 500 ppm
I20260620 00:07:35.265096 9693 webserver.cc:460] Webserver started at http://127.0.98.1:39229/ using document root <none> and password file <none>
I20260620 00:07:35.265409 9693 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:35.265484 9693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:35.265628 9693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:35.267160 9693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-0/data/instance:
uuid: "edbc7bf8bab44fc29d9d8e7638cac41e"
format_stamp: "Formatted at 2026-06-20 00:07:35 on dist-test-slave-grqt"
server_key: "7913c58df6003cb634a9c759ab61ad20"
server_key_iv: "65355354648c683b2f5972b45e1c938c"
server_key_version: "encryptionkey@0"
I20260620 00:07:35.267629 9693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-0/wal/instance:
uuid: "edbc7bf8bab44fc29d9d8e7638cac41e"
format_stamp: "Formatted at 2026-06-20 00:07:35 on dist-test-slave-grqt"
server_key: "7913c58df6003cb634a9c759ab61ad20"
server_key_iv: "65355354648c683b2f5972b45e1c938c"
server_key_version: "encryptionkey@0"
I20260620 00:07:35.269459 9693 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 00:07:35.270430 9707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:35.270629 9693 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:35.270720 9693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-0/wal
uuid: "edbc7bf8bab44fc29d9d8e7638cac41e"
format_stamp: "Formatted at 2026-06-20 00:07:35 on dist-test-slave-grqt"
server_key: "7913c58df6003cb634a9c759ab61ad20"
server_key_iv: "65355354648c683b2f5972b45e1c938c"
server_key_version: "encryptionkey@0"
I20260620 00:07:35.270802 9693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:35.302871 9693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:35.303237 9693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:35.303407 9693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:35.303742 9693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:07:35.304090 9693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:07:35.304126 9693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:35.304152 9693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:07:35.304163 9693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:35.310192 9693 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46027
I20260620 00:07:35.310271 9820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46027 every 8 connection(s)
I20260620 00:07:35.310596 9693 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:07:35.315917 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:35.311355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47959 (local address 127.0.98.62:36181)
0620 00:07:35.311399 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:35.311401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:35.311669 (+ 268us) server_negotiation.cc:407] Connection header received
0620 00:07:35.312214 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:35.312215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:35.312231 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:35.312281 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:35.312958 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:35.313495 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:35.314504 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:35.314704 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:35.314782 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:35.315623 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:35.315624 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:35.315625 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:07:35.315629 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:35.315651 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:07:35.315655 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:35.315716 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:35.315811 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 00:07:35.315836 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 00:07:35.316190 9823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:35.311332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36181 (local address 127.0.98.1:47959)
0620 00:07:35.311543 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:35.311578 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:35.312105 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:35.312371 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:35.312375 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:35.312383 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:35.312857 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:35.312861 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:35.313595 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:35.313596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:35.314392 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:35.314395 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:35.314789 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:35.314790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:35.314923 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:35.315470 (+ 547us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:07:35.315470 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:35.315499 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:07:35.315523 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:35.315716 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:35.315722 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:35.315725 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:07:35.315813 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 00:07:35.315916 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":80,"threads_started":1}
I20260620 00:07:35.316774 9821 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36181
I20260620 00:07:35.316881 9821 heartbeater.cc:464] Registering TS with master...
I20260620 00:07:35.317051 9821 heartbeater.cc:511] Master 127.0.98.62:36181 requested a full tablet report, sending...
I20260620 00:07:35.317667 9633 ts_manager.cc:194] Registered new tserver with Master: edbc7bf8bab44fc29d9d8e7638cac41e (127.0.98.1:46027)
I20260620 00:07:35.318316 9633 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:47959
I20260620 00:07:35.319278 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 9693
I20260620 00:07:35.319372 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:07:35.319562 392 external_mini_cluster.cc:1428] Setting key 5339efa7dc2a169c1e83ed73814b870a
I20260620 00:07:35.320786 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36181
--builtin_ntp_servers=127.0.98.20:43403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 00:07:35.403055 9824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:35.403261 9824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:35.403283 9824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:35.404385 9824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:35.404417 9824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:35.404441 9824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:07:35.405862 9824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:35.406064 9824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:35.406301 9824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:35.411304 9832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:35.411324 9829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:35.411427 9830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:35.411499 9824 server_base.cc:1034] running on GCE node
I20260620 00:07:35.411934 9824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:35.412190 9824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:35.413354 9824 hybrid_clock.cc:648] HybridClock initialized: now 1781914055413355 us; error 44 us; skew 500 ppm
I20260620 00:07:35.414685 9824 webserver.cc:460] Webserver started at http://127.0.98.2:41417/ using document root <none> and password file <none>
I20260620 00:07:35.414933 9824 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:35.415011 9824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:35.415158 9824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:35.416219 9824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-1/data/instance:
uuid: "78b4b2b28a5c44f1b6f133d1b967068a"
format_stamp: "Formatted at 2026-06-20 00:07:35 on dist-test-slave-grqt"
server_key: "c7c9a834b187363003305c2eeacd6cdd"
server_key_iv: "b28caf45cf2e11f55a5e2af8412f6f84"
server_key_version: "encryptionkey@0"
I20260620 00:07:35.416582 9824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-1/wal/instance:
uuid: "78b4b2b28a5c44f1b6f133d1b967068a"
format_stamp: "Formatted at 2026-06-20 00:07:35 on dist-test-slave-grqt"
server_key: "c7c9a834b187363003305c2eeacd6cdd"
server_key_iv: "b28caf45cf2e11f55a5e2af8412f6f84"
server_key_version: "encryptionkey@0"
I20260620 00:07:35.417968 9824 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:07:35.418681 9838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:35.418854 9824 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 00:07:35.418956 9824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-1/wal
uuid: "78b4b2b28a5c44f1b6f133d1b967068a"
format_stamp: "Formatted at 2026-06-20 00:07:35 on dist-test-slave-grqt"
server_key: "c7c9a834b187363003305c2eeacd6cdd"
server_key_iv: "b28caf45cf2e11f55a5e2af8412f6f84"
server_key_version: "encryptionkey@0"
I20260620 00:07:35.419031 9824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:35.432524 9824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:35.432863 9824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:35.433023 9824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:35.433256 9824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:07:35.433624 9824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:07:35.433681 9824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:35.433723 9824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:07:35.433753 9824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:35.440676 9824 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39053
I20260620 00:07:35.440719 9951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39053 every 8 connection(s)
I20260620 00:07:35.441061 9824 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:07:35.445881 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 9824
I20260620 00:07:35.445992 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:07:35.446190 392 external_mini_cluster.cc:1428] Setting key ede3821e9bad1c1a291a7604c0e746f7
I20260620 00:07:35.447021 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:35.441850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40899 (local address 127.0.98.62:36181)
0620 00:07:35.441927 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:35.441930 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:35.442243 (+ 313us) server_negotiation.cc:407] Connection header received
0620 00:07:35.442936 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:35.442937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:35.442947 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:35.443008 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:35.443745 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:35.444260 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:35.445297 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:35.445613 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:35.445729 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:35.446759 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:35.446761 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:35.446762 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:07:35.446766 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:35.446781 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:07:35.446785 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:35.446834 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:35.446913 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 00:07:35.446939 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 00:07:35.447259 9954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:35.441821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36181 (local address 127.0.98.2:40899)
0620 00:07:35.442088 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:35.442123 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:35.442805 (+ 682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:35.443084 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:35.443088 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:35.443096 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:35.443632 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:35.443636 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:35.444351 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:35.444352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:35.445200 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:35.445202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:35.445757 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:35.445758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:35.445891 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:35.446585 (+ 694us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:07:35.446586 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:35.446626 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:07:35.446659 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:35.446834 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:35.446839 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:35.446842 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:07:35.446907 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 00:07:35.447014 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":111,"threads_started":1}
I20260620 00:07:35.447289 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36181
--builtin_ntp_servers=127.0.98.20:43403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 00:07:35.447758 9952 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36181
I20260620 00:07:35.447846 9952 heartbeater.cc:464] Registering TS with master...
I20260620 00:07:35.448048 9952 heartbeater.cc:511] Master 127.0.98.62:36181 requested a full tablet report, sending...
I20260620 00:07:35.448442 9633 ts_manager.cc:194] Registered new tserver with Master: 78b4b2b28a5c44f1b6f133d1b967068a (127.0.98.2:39053)
I20260620 00:07:35.448825 9633 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:40899
W20260620 00:07:35.528484 9955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:35.528810 9955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:35.528834 9955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:35.530120 9955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:35.530164 9955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:35.530190 9955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:07:35.531477 9955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:35.531678 9955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:35.531942 9955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:35.537278 9960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:35.537273 9961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:35.537273 9963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:35.537467 9955 server_base.cc:1034] running on GCE node
I20260620 00:07:35.537832 9955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:35.538064 9955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:35.539209 9955 hybrid_clock.cc:648] HybridClock initialized: now 1781914055539197 us; error 27 us; skew 500 ppm
I20260620 00:07:35.540516 9955 webserver.cc:460] Webserver started at http://127.0.98.3:41557/ using document root <none> and password file <none>
I20260620 00:07:35.540784 9955 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:35.540831 9955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:35.540987 9955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:35.542181 9955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-2/data/instance:
uuid: "9227780a78d9407e907a37cf62bfbc78"
format_stamp: "Formatted at 2026-06-20 00:07:35 on dist-test-slave-grqt"
server_key: "a1f9e20f35ee29ea645b1225dea13cd3"
server_key_iv: "b51a7dc62121425d14fd850ee5848bb0"
server_key_version: "encryptionkey@0"
I20260620 00:07:35.542551 9955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9227780a78d9407e907a37cf62bfbc78"
format_stamp: "Formatted at 2026-06-20 00:07:35 on dist-test-slave-grqt"
server_key: "a1f9e20f35ee29ea645b1225dea13cd3"
server_key_iv: "b51a7dc62121425d14fd850ee5848bb0"
server_key_version: "encryptionkey@0"
I20260620 00:07:35.543845 9955 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 00:07:35.544596 9969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:35.544763 9955 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:07:35.544862 9955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-2/wal
uuid: "9227780a78d9407e907a37cf62bfbc78"
format_stamp: "Formatted at 2026-06-20 00:07:35 on dist-test-slave-grqt"
server_key: "a1f9e20f35ee29ea645b1225dea13cd3"
server_key_iv: "b51a7dc62121425d14fd850ee5848bb0"
server_key_version: "encryptionkey@0"
I20260620 00:07:35.544945 9955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:35.554915 9955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:35.555254 9955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:35.555413 9955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:35.555655 9955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:07:35.555995 9955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:07:35.556051 9955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:35.556094 9955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:07:35.556128 9955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:35.562445 9955 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37563
I20260620 00:07:35.562496 10082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37563 every 8 connection(s)
I20260620 00:07:35.562855 9955 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:07:35.567713 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:35.563623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39213 (local address 127.0.98.62:36181)
0620 00:07:35.563698 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:35.563700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:35.563922 (+ 222us) server_negotiation.cc:407] Connection header received
0620 00:07:35.564396 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:35.564396 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:35.564407 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:35.564461 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:35.565089 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:35.565601 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:35.566525 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:35.566701 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:35.566754 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:35.567506 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:35.567507 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:35.567507 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:07:35.567512 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:35.567526 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:07:35.567530 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:35.567572 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:35.567633 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 00:07:35.567653 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 00:07:35.567936 10085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:35.563614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36181 (local address 127.0.98.3:39213)
0620 00:07:35.563801 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:35.563836 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:35.564309 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:35.564520 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:35.564523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:35.564531 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:35.565017 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:35.565021 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:35.565676 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:35.565677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:35.566443 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:35.566444 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:35.566772 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:35.566773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:35.566874 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:35.567370 (+ 496us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:07:35.567371 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:35.567405 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:07:35.567426 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:35.567572 (+ 146us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:35.567574 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:35.567577 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:07:35.567624 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 00:07:35.567699 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":97,"threads_started":1}
I20260620 00:07:35.568455 10083 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36181
I20260620 00:07:35.568575 10083 heartbeater.cc:464] Registering TS with master...
I20260620 00:07:35.568754 10083 heartbeater.cc:511] Master 127.0.98.62:36181 requested a full tablet report, sending...
I20260620 00:07:35.569083 9633 ts_manager.cc:194] Registered new tserver with Master: 9227780a78d9407e907a37cf62bfbc78 (127.0.98.3:37563)
I20260620 00:07:35.569533 9633 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:39213
I20260620 00:07:35.572640 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 9955
I20260620 00:07:35.572711 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914001714546-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:07:35.572916 392 external_mini_cluster.cc:1428] Setting key 8bd3c8251fc403c04e71380ff48b16f9
I20260620 00:07:35.573770 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 00:07:35.575487 9673 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:35.575080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56750 (local address 127.0.98.62:36181)
0620 00:07:35.575117 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:35.575119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:35.575236 (+ 117us) server_negotiation.cc:407] Connection header received
0620 00:07:35.575285 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:35.575286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:35.575293 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:35.575336 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:35.575438 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56750: BlockingRecv error: recv got EOF from 127.0.0.1:56750 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 00:07:35.575637 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 9693
I20260620 00:07:35.580602 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 9824
I20260620 00:07:35.584990 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 9955
I20260620 00:07:35.589100 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 9602
2026-06-20T00:07:35Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (632 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T00:07:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:07:35Z Disabled control of system clock
I20260620 00:07:35.616492 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37459
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 00:07:35.699888 10094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:35.700121 10094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:35.700143 10094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:35.701356 10094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:07:35.701473 10094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:35.701494 10094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:35.701508 10094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:07:35.701521 10094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:07:35.702886 10094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37459
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:35.703099 10094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:35.703346 10094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:35.708340 10099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:35.708408 10094 server_base.cc:1034] running on GCE node
W20260620 00:07:35.708349 10100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:35.708348 10102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:35.708853 10094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:35.709192 10094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:35.710392 10094 hybrid_clock.cc:648] HybridClock initialized: now 1781914055710386 us; error 66 us; skew 500 ppm
I20260620 00:07:35.711683 10094 webserver.cc:460] Webserver started at http://127.0.98.62:40941/ using document root <none> and password file <none>
I20260620 00:07:35.711951 10094 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:35.712026 10094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:35.712178 10094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:35.713285 10094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "1050932a644e476a88cdb9db35d77733"
format_stamp: "Formatted at 2026-06-20 00:07:35 on dist-test-slave-grqt"
server_key: "7634bc8aa8706b61955edbc3384df154"
server_key_iv: "1c81f23896979b25e9598f25a34e6997"
server_key_version: "encryptionkey@0"
I20260620 00:07:35.713735 10094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "1050932a644e476a88cdb9db35d77733"
format_stamp: "Formatted at 2026-06-20 00:07:35 on dist-test-slave-grqt"
server_key: "7634bc8aa8706b61955edbc3384df154"
server_key_iv: "1c81f23896979b25e9598f25a34e6997"
server_key_version: "encryptionkey@0"
I20260620 00:07:35.715112 10094 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:35.715888 10108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:35.716090 10094 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:07:35.716192 10094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "1050932a644e476a88cdb9db35d77733"
format_stamp: "Formatted at 2026-06-20 00:07:35 on dist-test-slave-grqt"
server_key: "7634bc8aa8706b61955edbc3384df154"
server_key_iv: "1c81f23896979b25e9598f25a34e6997"
server_key_version: "encryptionkey@0"
I20260620 00:07:35.716279 10094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:35.719791 10094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:35.720072 10094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:35.720219 10094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:35.724143 10094 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37459
I20260620 00:07:35.724179 10160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37459 every 8 connection(s)
I20260620 00:07:35.724522 10094 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:07:35.725318 10161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:35.728798 10161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1050932a644e476a88cdb9db35d77733: Bootstrap starting.
I20260620 00:07:35.729578 10161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1050932a644e476a88cdb9db35d77733: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:35.729919 10161 log.cc:824] T 00000000000000000000000000000000 P 1050932a644e476a88cdb9db35d77733: Log is configured to *not* fsync() on all Append() calls
I20260620 00:07:35.730765 10161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1050932a644e476a88cdb9db35d77733: No bootstrap required, opened a new log
I20260620 00:07:35.730938 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 10094
I20260620 00:07:35.731050 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:07:35.731249 392 external_mini_cluster.cc:1428] Setting key 5c1e96a0825a414bbf74f1e91267db7e
I20260620 00:07:35.732954 10161 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1050932a644e476a88cdb9db35d77733 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1050932a644e476a88cdb9db35d77733" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37459 } }
I20260620 00:07:35.733140 10161 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1050932a644e476a88cdb9db35d77733 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:35.733186 10161 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1050932a644e476a88cdb9db35d77733 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1050932a644e476a88cdb9db35d77733, State: Initialized, Role: FOLLOWER
I20260620 00:07:35.733310 10161 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1050932a644e476a88cdb9db35d77733 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1050932a644e476a88cdb9db35d77733" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37459 } }
I20260620 00:07:35.733434 10161 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1050932a644e476a88cdb9db35d77733 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:07:35.733474 10161 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1050932a644e476a88cdb9db35d77733 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:07:35.733518 10161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1050932a644e476a88cdb9db35d77733 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:35.734354 10161 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1050932a644e476a88cdb9db35d77733 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1050932a644e476a88cdb9db35d77733" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37459 } }
I20260620 00:07:35.734447 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:35.731900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53126 (local address 127.0.98.62:37459)
0620 00:07:35.732116 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:35.732123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:35.732148 (+ 25us) server_negotiation.cc:407] Connection header received
0620 00:07:35.732960 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:35.732969 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:35.733039 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:35.733217 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:35.733959 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:35.733963 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:35.733963 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:07:35.733968 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:35.734015 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:07:35.734029 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:35.734108 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:35.734169 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 00:07:35.734231 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":108,"threads_started":1}
I20260620 00:07:35.734465 10161 leader_election.cc:302] T 00000000000000000000000000000000 P 1050932a644e476a88cdb9db35d77733 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1050932a644e476a88cdb9db35d77733; no voters:
I20260620 00:07:35.734635 10161 leader_election.cc:288] T 00000000000000000000000000000000 P 1050932a644e476a88cdb9db35d77733 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:07:35.734917 10161 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1050932a644e476a88cdb9db35d77733 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:07:35.734935 10166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1050932a644e476a88cdb9db35d77733 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:07:35.735046 10166 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1050932a644e476a88cdb9db35d77733 [term 1 LEADER]: Becoming Leader. State: Replica: 1050932a644e476a88cdb9db35d77733, State: Running, Role: LEADER
I20260620 00:07:35.735143 10166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1050932a644e476a88cdb9db35d77733 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1050932a644e476a88cdb9db35d77733" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37459 } }
I20260620 00:07:35.735569 10168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1050932a644e476a88cdb9db35d77733 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1050932a644e476a88cdb9db35d77733. Latest consensus state: current_term: 1 leader_uuid: "1050932a644e476a88cdb9db35d77733" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1050932a644e476a88cdb9db35d77733" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37459 } } }
I20260620 00:07:35.735584 10167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1050932a644e476a88cdb9db35d77733 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1050932a644e476a88cdb9db35d77733" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1050932a644e476a88cdb9db35d77733" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37459 } } }
I20260620 00:07:35.735678 10168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1050932a644e476a88cdb9db35d77733 [sys.catalog]: This master's current role is: LEADER
I20260620 00:07:35.735689 10167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1050932a644e476a88cdb9db35d77733 [sys.catalog]: This master's current role is: LEADER
W20260620 00:07:35.736891 10182 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1050932a644e476a88cdb9db35d77733: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:07:35.736951 10182 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:07:35.737036 10175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:07:35.737236 10175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:07:35.738878 10175 catalog_manager.cc:1347] Generated new cluster ID: cd99223d9a214538ab523969a293bed4
I20260620 00:07:35.738937 10175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:07:35.750272 10175 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:07:35.750484 10175 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:07:35.759806 10175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1050932a644e476a88cdb9db35d77733: Generated new TSK 0
I20260620 00:07:35.760053 10175 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:07:35.762554 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37459
--builtin_ntp_servers=127.0.98.20:41077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 00:07:35.859926 10185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:35.860168 10185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:35.860205 10185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:35.861615 10185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:35.861666 10185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:35.861696 10185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:07:35.863199 10185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:35.863479 10185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:35.863854 10185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:35.869096 10190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:35.869087 10191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:35.869318 10185 server_base.cc:1034] running on GCE node
W20260620 00:07:35.869087 10193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:35.869733 10185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:35.870011 10185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:35.871188 10185 hybrid_clock.cc:648] HybridClock initialized: now 1781914055871172 us; error 33 us; skew 500 ppm
I20260620 00:07:35.872596 10185 webserver.cc:460] Webserver started at http://127.0.98.1:42447/ using document root <none> and password file <none>
I20260620 00:07:35.872862 10185 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:35.872937 10185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:35.873099 10185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:35.874421 10185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-0/data/instance:
uuid: "6682b3a538f940c4969bfa34f63af7fa"
format_stamp: "Formatted at 2026-06-20 00:07:35 on dist-test-slave-grqt"
server_key: "020858fd6508a0e00de47ecbc9a84f9b"
server_key_iv: "4b24d7a3280c8b19703636bbfc13fbc4"
server_key_version: "encryptionkey@0"
I20260620 00:07:35.874852 10185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6682b3a538f940c4969bfa34f63af7fa"
format_stamp: "Formatted at 2026-06-20 00:07:35 on dist-test-slave-grqt"
server_key: "020858fd6508a0e00de47ecbc9a84f9b"
server_key_iv: "4b24d7a3280c8b19703636bbfc13fbc4"
server_key_version: "encryptionkey@0"
I20260620 00:07:35.876299 10185 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:07:35.877172 10199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:35.877351 10185 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:35.877552 10185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-0/wal
uuid: "6682b3a538f940c4969bfa34f63af7fa"
format_stamp: "Formatted at 2026-06-20 00:07:35 on dist-test-slave-grqt"
server_key: "020858fd6508a0e00de47ecbc9a84f9b"
server_key_iv: "4b24d7a3280c8b19703636bbfc13fbc4"
server_key_version: "encryptionkey@0"
I20260620 00:07:35.877640 10185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:35.881172 10185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:35.881551 10185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:35.881716 10185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:35.881947 10185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:07:35.882290 10185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:07:35.882349 10185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:35.882391 10185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:07:35.882418 10185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:35.888815 10185 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44615
I20260620 00:07:35.888852 10312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44615 every 8 connection(s)
I20260620 00:07:35.889199 10185 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:07:35.892417 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:35.889958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60145 (local address 127.0.98.62:37459)
0620 00:07:35.890002 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:35.890006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:35.890301 (+ 295us) server_negotiation.cc:407] Connection header received
0620 00:07:35.890825 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:35.890826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:35.890834 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:35.890893 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:35.892144 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:35.892145 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:35.892146 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:07:35.892151 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:35.892167 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:07:35.892171 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:35.892231 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:35.892332 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 00:07:35.892349 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 00:07:35.892617 10315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:35.889952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37459 (local address 127.0.98.1:60145)
0620 00:07:35.890148 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:35.890207 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:35.890713 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:35.890971 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:35.890976 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:35.890986 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:35.891978 (+ 992us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:07:35.891978 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:35.892014 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:07:35.892043 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:35.892230 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:35.892236 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:35.892239 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:07:35.892332 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 00:07:35.892396 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":89,"threads_started":1}
I20260620 00:07:35.893123 10313 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37459
I20260620 00:07:35.893214 10313 heartbeater.cc:464] Registering TS with master...
I20260620 00:07:35.893378 10313 heartbeater.cc:511] Master 127.0.98.62:37459 requested a full tablet report, sending...
I20260620 00:07:35.894032 10125 ts_manager.cc:194] Registered new tserver with Master: 6682b3a538f940c4969bfa34f63af7fa (127.0.98.1:44615)
I20260620 00:07:35.898566 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 10185
I20260620 00:07:35.898648 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:07:35.898804 392 external_mini_cluster.cc:1428] Setting key 282272d74f228aca27ce54e1e38265b1
I20260620 00:07:35.899935 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37459
--builtin_ntp_servers=127.0.98.20:41077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 00:07:35.982604 10316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:35.982839 10316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:35.982867 10316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:35.984180 10316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:35.984237 10316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:35.984267 10316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:07:35.985792 10316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:35.986021 10316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:35.986281 10316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 00:07:35.992233 10316 server_base.cc:1034] running on GCE node
W20260620 00:07:35.992198 10322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:35.992193 10324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:35.992283 10321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:35.992973 10316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:35.993278 10316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:35.994500 10316 hybrid_clock.cc:648] HybridClock initialized: now 1781914055994491 us; error 60 us; skew 500 ppm
I20260620 00:07:35.996095 10316 webserver.cc:460] Webserver started at http://127.0.98.2:34255/ using document root <none> and password file <none>
I20260620 00:07:35.996377 10316 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:35.996448 10316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:35.996613 10316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:35.997874 10316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-1/data/instance:
uuid: "4a34addad7fe4082ac13f8e9675dcf10"
format_stamp: "Formatted at 2026-06-20 00:07:35 on dist-test-slave-grqt"
server_key: "6bb06ee6c04fd4959291cd2df6ac2570"
server_key_iv: "d1a50a8d5605a50ad724a7cdb7e358e9"
server_key_version: "encryptionkey@0"
I20260620 00:07:35.998265 10316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4a34addad7fe4082ac13f8e9675dcf10"
format_stamp: "Formatted at 2026-06-20 00:07:35 on dist-test-slave-grqt"
server_key: "6bb06ee6c04fd4959291cd2df6ac2570"
server_key_iv: "d1a50a8d5605a50ad724a7cdb7e358e9"
server_key_version: "encryptionkey@0"
I20260620 00:07:35.999928 10316 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:07:36.000934 10330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:36.001241 10316 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:36.001340 10316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-1/wal
uuid: "4a34addad7fe4082ac13f8e9675dcf10"
format_stamp: "Formatted at 2026-06-20 00:07:35 on dist-test-slave-grqt"
server_key: "6bb06ee6c04fd4959291cd2df6ac2570"
server_key_iv: "d1a50a8d5605a50ad724a7cdb7e358e9"
server_key_version: "encryptionkey@0"
I20260620 00:07:36.001457 10316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:36.004905 10316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:36.005220 10316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:36.005410 10316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:36.005647 10316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:07:36.005985 10316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:07:36.006040 10316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:36.006091 10316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:07:36.006119 10316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:36.012054 10316 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33967
I20260620 00:07:36.012096 10443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33967 every 8 connection(s)
I20260620 00:07:36.012458 10316 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:07:36.015017 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 10316
I20260620 00:07:36.015149 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:07:36.015388 392 external_mini_cluster.cc:1428] Setting key 419a44ccea65febfb8bbe707dc860f5a
I20260620 00:07:36.015967 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:36.013258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36965 (local address 127.0.98.62:37459)
0620 00:07:36.013314 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:36.013316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:36.013626 (+ 310us) server_negotiation.cc:407] Connection header received
0620 00:07:36.014078 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:36.014079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:36.014083 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:36.014148 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:36.015659 (+ 1511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:36.015660 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:36.015660 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:07:36.015665 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:36.015688 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:07:36.015692 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:36.015779 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:36.015878 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 00:07:36.015896 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 00:07:36.016227 10446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:36.013225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37459 (local address 127.0.98.2:36965)
0620 00:07:36.013483 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:36.013528 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:36.013998 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:36.014227 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:36.014230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:36.014238 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:36.015462 (+ 1224us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:07:36.015463 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:36.015508 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:07:36.015548 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:36.015778 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:36.015786 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:36.015789 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:07:36.015890 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 00:07:36.015980 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":103,"threads_started":1}
I20260620 00:07:36.016359 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37459
--builtin_ntp_servers=127.0.98.20:41077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 00:07:36.017839 10444 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37459
I20260620 00:07:36.018010 10444 heartbeater.cc:464] Registering TS with master...
I20260620 00:07:36.018187 10444 heartbeater.cc:511] Master 127.0.98.62:37459 requested a full tablet report, sending...
I20260620 00:07:36.018628 10125 ts_manager.cc:194] Registered new tserver with Master: 4a34addad7fe4082ac13f8e9675dcf10 (127.0.98.2:33967)
W20260620 00:07:36.101302 10447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:36.101616 10447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:36.101643 10447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:36.102999 10447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:36.103055 10447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:36.103083 10447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:07:36.104663 10447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:36.105100 10447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:36.105634 10447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:36.111786 10453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:36.111678 10455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:36.111680 10452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:36.112182 10447 server_base.cc:1034] running on GCE node
I20260620 00:07:36.112547 10447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:36.112828 10447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:36.114019 10447 hybrid_clock.cc:648] HybridClock initialized: now 1781914056114006 us; error 31 us; skew 500 ppm
I20260620 00:07:36.115723 10447 webserver.cc:460] Webserver started at http://127.0.98.3:43975/ using document root <none> and password file <none>
I20260620 00:07:36.116026 10447 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:36.116103 10447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:36.116261 10447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:36.117452 10447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-2/data/instance:
uuid: "0d0b6841746b48368d74178cfd1fc36c"
format_stamp: "Formatted at 2026-06-20 00:07:36 on dist-test-slave-grqt"
server_key: "0250d599a393688812755981f0a55f8b"
server_key_iv: "f65e7d630afb06b0d190c810d34d8394"
server_key_version: "encryptionkey@0"
I20260620 00:07:36.117838 10447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0d0b6841746b48368d74178cfd1fc36c"
format_stamp: "Formatted at 2026-06-20 00:07:36 on dist-test-slave-grqt"
server_key: "0250d599a393688812755981f0a55f8b"
server_key_iv: "f65e7d630afb06b0d190c810d34d8394"
server_key_version: "encryptionkey@0"
I20260620 00:07:36.119246 10447 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 00:07:36.120589 10461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:36.120868 10447 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:07:36.120994 10447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-2/wal
uuid: "0d0b6841746b48368d74178cfd1fc36c"
format_stamp: "Formatted at 2026-06-20 00:07:36 on dist-test-slave-grqt"
server_key: "0250d599a393688812755981f0a55f8b"
server_key_iv: "f65e7d630afb06b0d190c810d34d8394"
server_key_version: "encryptionkey@0"
I20260620 00:07:36.121083 10447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:36.124755 10447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:36.125061 10447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:36.125208 10447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:36.125474 10447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:07:36.125805 10447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:07:36.125860 10447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:36.125901 10447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:07:36.125928 10447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:36.132920 10447 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33595
I20260620 00:07:36.132978 10574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33595 every 8 connection(s)
I20260620 00:07:36.133456 10447 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:07:36.136777 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:36.134322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59267 (local address 127.0.98.62:37459)
0620 00:07:36.134386 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:36.134388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:36.134583 (+ 195us) server_negotiation.cc:407] Connection header received
0620 00:07:36.135120 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:36.135121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:36.135126 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:36.135190 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:36.136512 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:36.136514 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:36.136514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:07:36.136518 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:36.136536 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:07:36.136541 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:36.136601 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:36.136703 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 00:07:36.136716 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 00:07:36.137085 10577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:36.134229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37459 (local address 127.0.98.3:59267)
0620 00:07:36.134442 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:36.134487 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:36.135016 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:36.135250 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:36.135259 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:36.135267 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:36.136323 (+ 1056us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:07:36.136324 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:36.136360 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:07:36.136396 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:36.136617 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:36.136621 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:36.136624 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:07:36.136714 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 00:07:36.136774 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":98,"threads_started":1}
I20260620 00:07:36.138031 10575 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37459
I20260620 00:07:36.138132 10575 heartbeater.cc:464] Registering TS with master...
I20260620 00:07:36.138316 10575 heartbeater.cc:511] Master 127.0.98.62:37459 requested a full tablet report, sending...
I20260620 00:07:36.139163 10125 ts_manager.cc:194] Registered new tserver with Master: 0d0b6841746b48368d74178cfd1fc36c (127.0.98.3:33595)
I20260620 00:07:36.142041 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 10447
I20260620 00:07:36.142136 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914001714546-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:07:36.142299 392 external_mini_cluster.cc:1428] Setting key 287affb389b942a2385f73abda8f75a1
I20260620 00:07:36.143745 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 00:07:36.146005 10165 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:07:36.145157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53140 (local address 127.0.98.62:37459)
0620 00:07:36.145272 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:36.145275 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:36.145402 (+ 127us) server_negotiation.cc:407] Connection header received
0620 00:07:36.145687 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:36.145688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:36.145695 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:36.145761 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:36.145901 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53140: BlockingRecv error: recv got EOF from 127.0.0.1:53140 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 00:07:36.146245 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 10185
I20260620 00:07:36.151635 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 10316
I20260620 00:07:36.156641 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 10447
I20260620 00:07:36.161147 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 10094
2026-06-20T00:07:36Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (573 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:07:36 dist-test-slave-grqt krb5kdc[10585](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:07:36 dist-test-slave-grqt krb5kdc[10585](info): set up 2 sockets
Jun 20 00:07:36 dist-test-slave-grqt krb5kdc[10585](info): commencing operation
krb5kdc: starting...
W20260620 00:07:37.238373 392 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:07:37 dist-test-slave-grqt krb5kdc[10585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:07:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:07:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:07:37.364437 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41809
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41809
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:07:37.446903 10598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:37.447134 10598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:37.447158 10598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:37.448434 10598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:07:37.448499 10598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:37.448514 10598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:37.448529 10598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:07:37.448540 10598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:07:37.450264 10598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:37.450618 10598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:37.450888 10598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:37.457062 10603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:37.457041 10606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:37.457041 10604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:37.457543 10598 server_base.cc:1034] running on GCE node
I20260620 00:07:37.457865 10598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:37.458161 10598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:37.459337 10598 hybrid_clock.cc:648] HybridClock initialized: now 1781914057459278 us; error 73 us; skew 500 ppm
Jun 20 00:07:37 dist-test-slave-grqt krb5kdc[10585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:37.462452 10598 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:07:37.462950 10598 webserver.cc:460] Webserver started at http://127.0.98.62:42591/ using document root <none> and password file <none>
I20260620 00:07:37.463199 10598 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:37.463251 10598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:37.463402 10598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:37.464533 10598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "1000dfe1dc264e16aeb9e817801e380d"
format_stamp: "Formatted at 2026-06-20 00:07:37 on dist-test-slave-grqt"
server_key: "f60c93609c7aa55164c9de633b47777c"
server_key_iv: "8216511b3335af2a01ada483df9ecd98"
server_key_version: "encryptionkey@0"
I20260620 00:07:37.464905 10598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "1000dfe1dc264e16aeb9e817801e380d"
format_stamp: "Formatted at 2026-06-20 00:07:37 on dist-test-slave-grqt"
server_key: "f60c93609c7aa55164c9de633b47777c"
server_key_iv: "8216511b3335af2a01ada483df9ecd98"
server_key_version: "encryptionkey@0"
I20260620 00:07:37.466655 10598 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 00:07:37.467700 10613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:37.467999 10598 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:37.468123 10598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "1000dfe1dc264e16aeb9e817801e380d"
format_stamp: "Formatted at 2026-06-20 00:07:37 on dist-test-slave-grqt"
server_key: "f60c93609c7aa55164c9de633b47777c"
server_key_iv: "8216511b3335af2a01ada483df9ecd98"
server_key_version: "encryptionkey@0"
I20260620 00:07:37.468214 10598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:37.484319 10598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:37.486944 10598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:37.487116 10598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:37.491060 10598 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41809
I20260620 00:07:37.491127 10665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41809 every 8 connection(s)
I20260620 00:07:37.491487 10598 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:07:37.492551 10666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:37.494987 10666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1000dfe1dc264e16aeb9e817801e380d: Bootstrap starting.
I20260620 00:07:37.495615 10666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1000dfe1dc264e16aeb9e817801e380d: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:37.495932 10666 log.cc:824] T 00000000000000000000000000000000 P 1000dfe1dc264e16aeb9e817801e380d: Log is configured to *not* fsync() on all Append() calls
I20260620 00:07:37.496690 10666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1000dfe1dc264e16aeb9e817801e380d: No bootstrap required, opened a new log
I20260620 00:07:37.498517 10666 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1000dfe1dc264e16aeb9e817801e380d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1000dfe1dc264e16aeb9e817801e380d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41809 } }
I20260620 00:07:37.498683 10666 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1000dfe1dc264e16aeb9e817801e380d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:37.498739 10666 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1000dfe1dc264e16aeb9e817801e380d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1000dfe1dc264e16aeb9e817801e380d, State: Initialized, Role: FOLLOWER
I20260620 00:07:37.498914 10666 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1000dfe1dc264e16aeb9e817801e380d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1000dfe1dc264e16aeb9e817801e380d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41809 } }
I20260620 00:07:37.499003 10666 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1000dfe1dc264e16aeb9e817801e380d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:07:37.499029 10666 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1000dfe1dc264e16aeb9e817801e380d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:07:37.499099 10666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1000dfe1dc264e16aeb9e817801e380d [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:37.499634 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 10598
I20260620 00:07:37.499750 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:07:37.499974 392 external_mini_cluster.cc:1428] Setting key dc26b94ab6508f7b4ee3f449116d5d56
I20260620 00:07:37.500270 10666 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1000dfe1dc264e16aeb9e817801e380d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1000dfe1dc264e16aeb9e817801e380d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41809 } }
I20260620 00:07:37.500430 10666 leader_election.cc:302] T 00000000000000000000000000000000 P 1000dfe1dc264e16aeb9e817801e380d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1000dfe1dc264e16aeb9e817801e380d; no voters:
I20260620 00:07:37.500646 10666 leader_election.cc:288] T 00000000000000000000000000000000 P 1000dfe1dc264e16aeb9e817801e380d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:07:37.500824 10672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1000dfe1dc264e16aeb9e817801e380d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:07:37.500946 10666 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1000dfe1dc264e16aeb9e817801e380d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:07:37.500991 10672 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1000dfe1dc264e16aeb9e817801e380d [term 1 LEADER]: Becoming Leader. State: Replica: 1000dfe1dc264e16aeb9e817801e380d, State: Running, Role: LEADER
I20260620 00:07:37.501119 10672 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1000dfe1dc264e16aeb9e817801e380d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1000dfe1dc264e16aeb9e817801e380d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41809 } }
I20260620 00:07:37.501663 10673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1000dfe1dc264e16aeb9e817801e380d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1000dfe1dc264e16aeb9e817801e380d. Latest consensus state: current_term: 1 leader_uuid: "1000dfe1dc264e16aeb9e817801e380d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1000dfe1dc264e16aeb9e817801e380d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41809 } } }
I20260620 00:07:37.501823 10673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1000dfe1dc264e16aeb9e817801e380d [sys.catalog]: This master's current role is: LEADER
I20260620 00:07:37.502089 10680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:07:37.502204 10671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1000dfe1dc264e16aeb9e817801e380d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1000dfe1dc264e16aeb9e817801e380d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1000dfe1dc264e16aeb9e817801e380d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41809 } } }
I20260620 00:07:37.502331 10671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1000dfe1dc264e16aeb9e817801e380d [sys.catalog]: This master's current role is: LEADER
I20260620 00:07:37.502779 10680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:07:37.504593 10680 catalog_manager.cc:1347] Generated new cluster ID: a212913ea0284809971179e429060991
I20260620 00:07:37.504660 10680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 00:07:37 dist-test-slave-grqt krb5kdc[10585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:07:37.512571 10670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:37.500499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57162 (local address 127.0.98.62:41809)
0620 00:07:37.500729 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:37.500741 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:37.500790 (+ 49us) server_negotiation.cc:407] Connection header received
0620 00:07:37.500930 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:37.500940 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:37.501004 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:37.501146 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:37.501961 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:37.503412 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:37.504369 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:37.504970 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:37.505092 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:37.508063 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:37.508068 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:37.508069 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:37.508083 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:37.510809 (+ 2726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:37.511236 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:37.511240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:37.511240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:37.511376 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:37.511586 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:37.511588 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:37.511588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:37.511940 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:37.512111 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:37.512186 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 00:07:37.512335 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":110,"threads_started":1}
I20260620 00:07:37.524542 10680 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:07:37.525425 10680 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:07:37.532626 10680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1000dfe1dc264e16aeb9e817801e380d: Generated new TSK 0
I20260620 00:07:37.532851 10680 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:07:37.569151 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41809
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:07:37.684528 10692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:37.684755 10692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:37.684782 10692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:37.686077 10692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:37.686127 10692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:37.686156 10692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:07:37.687623 10692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:37.687827 10692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:37.688014 10692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:37.693056 10700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:37.693085 10697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:37.693125 10698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:37.693465 10692 server_base.cc:1034] running on GCE node
I20260620 00:07:37.693691 10692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:37.693951 10692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:37.695147 10692 hybrid_clock.cc:648] HybridClock initialized: now 1781914057695128 us; error 32 us; skew 500 ppm
Jun 20 00:07:37 dist-test-slave-grqt krb5kdc[10585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:37.697786 10692 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:07:37.698278 10692 webserver.cc:460] Webserver started at http://127.0.98.1:42533/ using document root <none> and password file <none>
I20260620 00:07:37.698719 10692 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:37.698779 10692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:37.698920 10692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:37.700018 10692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-0/data/instance:
uuid: "068a3d41f2ca4bff9d581eac08ac19c4"
format_stamp: "Formatted at 2026-06-20 00:07:37 on dist-test-slave-grqt"
server_key: "7732b29e3237898322e8debe358166ac"
server_key_iv: "bdf0b520c5926e358ed198239de89da7"
server_key_version: "encryptionkey@0"
I20260620 00:07:37.700397 10692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-0/wal/instance:
uuid: "068a3d41f2ca4bff9d581eac08ac19c4"
format_stamp: "Formatted at 2026-06-20 00:07:37 on dist-test-slave-grqt"
server_key: "7732b29e3237898322e8debe358166ac"
server_key_iv: "bdf0b520c5926e358ed198239de89da7"
server_key_version: "encryptionkey@0"
I20260620 00:07:37.701941 10692 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:07:37.702886 10707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:37.703110 10692 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:37.703207 10692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-0/wal
uuid: "068a3d41f2ca4bff9d581eac08ac19c4"
format_stamp: "Formatted at 2026-06-20 00:07:37 on dist-test-slave-grqt"
server_key: "7732b29e3237898322e8debe358166ac"
server_key_iv: "bdf0b520c5926e358ed198239de89da7"
server_key_version: "encryptionkey@0"
I20260620 00:07:37.703289 10692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:37.720659 10692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:37.723268 10692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:37.723428 10692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:37.723694 10692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:07:37.724045 10692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:07:37.724107 10692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:37.724150 10692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:07:37.724177 10692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:37.730902 10692 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36921
I20260620 00:07:37.730939 10820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36921 every 8 connection(s)
I20260620 00:07:37.731288 10692 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:07:37.735893 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 10692
I20260620 00:07:37.736019 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:07:37.736261 392 external_mini_cluster.cc:1428] Setting key 5d1898b4181da3a908c2f4941fab4c86
Jun 20 00:07:37 dist-test-slave-grqt krb5kdc[10585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:07:37.743685 10670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:37.732249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54131 (local address 127.0.98.62:41809)
0620 00:07:37.732339 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:37.732341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:37.732530 (+ 189us) server_negotiation.cc:407] Connection header received
0620 00:07:37.733013 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:37.733015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:37.733028 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:37.733097 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:37.734353 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:37.734991 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:37.736191 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:37.736495 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:37.736611 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:37.739905 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:37.739907 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:37.739907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:37.739915 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:37.741870 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:37.742481 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:37.742482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:37.742482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:37.742539 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:37.742846 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:37.742847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:37.742847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:37.742965 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:37.743018 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:37.743517 (+ 499us) server_negotiation.cc:299] Negotiation successful
0620 00:07:37.743595 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 00:07:37.743930 10823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:37.732185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41809 (local address 127.0.98.1:54131)
0620 00:07:37.732368 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:37.732411 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:37.732900 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:37.733214 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:37.733218 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:37.733927 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:37.734143 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:37.734148 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:37.735103 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:37.735104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:37.736087 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:37.736091 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:37.736639 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:37.736640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:37.736791 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:37.737518 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:37.737527 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:37.739780 (+ 2253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:37.741976 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:37.741986 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:37.741990 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:37.742377 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:37.742614 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:37.742615 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:37.742616 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:37.742776 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:37.743035 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:37.743042 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:37.743239 (+ 197us) client_negotiation.cc:769] Sending connection context
0620 00:07:37.743370 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 00:07:37.743580 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
I20260620 00:07:37.744699 10821 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41809
I20260620 00:07:37.744817 10821 heartbeater.cc:464] Registering TS with master...
I20260620 00:07:37.745029 10821 heartbeater.cc:511] Master 127.0.98.62:41809 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 00:07:37.745716 10630 ts_manager.cc:194] Registered new tserver with Master: 068a3d41f2ca4bff9d581eac08ac19c4 (127.0.98.1:36921)
I20260620 00:07:37.746400 10630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54131
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:07:37.765632 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41809
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:07:37.848652 10826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:37.848898 10826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:37.848927 10826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:37.850260 10826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:37.850315 10826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:37.850346 10826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:07:37.851960 10826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:37.852250 10826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:37.852541 10826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:37.858791 10831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:37.858806 10832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:37.858791 10834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:37.859087 10826 server_base.cc:1034] running on GCE node
I20260620 00:07:37.859300 10826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:37.859593 10826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:37.860790 10826 hybrid_clock.cc:648] HybridClock initialized: now 1781914057860741 us; error 73 us; skew 500 ppm
Jun 20 00:07:37 dist-test-slave-grqt krb5kdc[10585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:37.863853 10826 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:07:37.864387 10826 webserver.cc:460] Webserver started at http://127.0.98.2:40051/ using document root <none> and password file <none>
I20260620 00:07:37.864599 10826 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:37.864643 10826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:37.864738 10826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:37.865886 10826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-1/data/instance:
uuid: "a252b24e99b94769b31d9a50d12dc538"
format_stamp: "Formatted at 2026-06-20 00:07:37 on dist-test-slave-grqt"
server_key: "8c5b9e98f44cb02772fcd083454a13a0"
server_key_iv: "13316ceec05f3ce8f1bbe14aa737995e"
server_key_version: "encryptionkey@0"
I20260620 00:07:37.866196 10826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a252b24e99b94769b31d9a50d12dc538"
format_stamp: "Formatted at 2026-06-20 00:07:37 on dist-test-slave-grqt"
server_key: "8c5b9e98f44cb02772fcd083454a13a0"
server_key_iv: "13316ceec05f3ce8f1bbe14aa737995e"
server_key_version: "encryptionkey@0"
I20260620 00:07:37.867528 10826 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:07:37.868294 10841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:37.868498 10826 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:37.868584 10826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-1/wal
uuid: "a252b24e99b94769b31d9a50d12dc538"
format_stamp: "Formatted at 2026-06-20 00:07:37 on dist-test-slave-grqt"
server_key: "8c5b9e98f44cb02772fcd083454a13a0"
server_key_iv: "13316ceec05f3ce8f1bbe14aa737995e"
server_key_version: "encryptionkey@0"
I20260620 00:07:37.868641 10826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:37.884945 10826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:37.887423 10826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:37.887547 10826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:37.887745 10826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:07:37.888118 10826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:07:37.888156 10826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:37.888180 10826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:07:37.888191 10826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:37.894166 10826 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44117
I20260620 00:07:37.894243 10954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44117 every 8 connection(s)
I20260620 00:07:37.894567 10826 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:07:37.900955 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 10826
I20260620 00:07:37.901057 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-1/wal/instance
Jun 20 00:07:37 dist-test-slave-grqt krb5kdc[10585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:07:37.902274 392 external_mini_cluster.cc:1428] Setting key a671b4b2de669a0d58d6faa96f60398a
I20260620 00:07:37.905443 10670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:37.895345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34173 (local address 127.0.98.62:41809)
0620 00:07:37.895440 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:37.895443 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:37.895589 (+ 146us) server_negotiation.cc:407] Connection header received
0620 00:07:37.896589 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:37.896590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:37.896603 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:37.896653 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:37.897322 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:37.897877 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:37.898904 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:37.899106 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:37.899192 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:37.902066 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:37.902067 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:37.902068 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:37.902078 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:37.903625 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:37.904155 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:37.904156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:37.904156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:37.904241 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:37.904572 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:37.904573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:37.904573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:37.904769 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:37.904849 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:37.905198 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 00:07:37.905298 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 00:07:37.905660 10957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:37.895275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41809 (local address 127.0.98.2:34173)
0620 00:07:37.895469 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:37.895512 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:37.896465 (+ 953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:37.896737 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:37.896741 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:37.897032 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:37.897220 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:37.897224 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:37.897970 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:37.897971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:37.898792 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:37.898794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:37.899202 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:37.899203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:37.899343 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:37.899930 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:37.899939 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:37.901926 (+ 1987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:37.903714 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:37.903721 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:37.903723 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:37.904074 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:37.904331 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:37.904332 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:37.904333 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:37.904500 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:37.904899 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:37.904906 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:37.905051 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 00:07:37.905153 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 00:07:37.905299 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":91,"threads_started":1}
I20260620 00:07:37.906132 10955 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41809
I20260620 00:07:37.906255 10955 heartbeater.cc:464] Registering TS with master...
I20260620 00:07:37.906440 10955 heartbeater.cc:511] Master 127.0.98.62:41809 requested a full tablet report, sending...
I20260620 00:07:37.906852 10630 ts_manager.cc:194] Registered new tserver with Master: a252b24e99b94769b31d9a50d12dc538 (127.0.98.2:44117)
I20260620 00:07:37.907322 10630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34173
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:07:37.928968 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41809
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:07:38.013777 10960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:07:38.014030 10960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:07:38.014070 10960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:07:38.015277 10960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:07:38.015348 10960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:07:38.015395 10960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:07:38.016937 10960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:07:38.017217 10960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:07:38.017524 10960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:07:38.022863 10968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:38.022867 10965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:07:38.022867 10966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:07:38.023136 10960 server_base.cc:1034] running on GCE node
I20260620 00:07:38.023370 10960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:07:38.023627 10960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:07:38.024782 10960 hybrid_clock.cc:648] HybridClock initialized: now 1781914058024769 us; error 29 us; skew 500 ppm
Jun 20 00:07:38 dist-test-slave-grqt krb5kdc[10585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914058, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:07:38.027518 10960 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:07:38.027925 10960 webserver.cc:460] Webserver started at http://127.0.98.3:39163/ using document root <none> and password file <none>
I20260620 00:07:38.028108 10960 fs_manager.cc:359] Metadata directory not provided
I20260620 00:07:38.028149 10960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:07:38.028240 10960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:07:38.029249 10960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-2/data/instance:
uuid: "58fb7c9f8e3c4924b76c6ae3a40f9ce4"
format_stamp: "Formatted at 2026-06-20 00:07:38 on dist-test-slave-grqt"
server_key: "b24879e6c68c7d847258006824f44bd1"
server_key_iv: "79f1d268532b0c679fd9035b4ff0e6b8"
server_key_version: "encryptionkey@0"
I20260620 00:07:38.029645 10960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-2/wal/instance:
uuid: "58fb7c9f8e3c4924b76c6ae3a40f9ce4"
format_stamp: "Formatted at 2026-06-20 00:07:38 on dist-test-slave-grqt"
server_key: "b24879e6c68c7d847258006824f44bd1"
server_key_iv: "79f1d268532b0c679fd9035b4ff0e6b8"
server_key_version: "encryptionkey@0"
I20260620 00:07:38.030956 10960 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:38.031682 10975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:38.031862 10960 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:07:38.031970 10960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-2/wal
uuid: "58fb7c9f8e3c4924b76c6ae3a40f9ce4"
format_stamp: "Formatted at 2026-06-20 00:07:38 on dist-test-slave-grqt"
server_key: "b24879e6c68c7d847258006824f44bd1"
server_key_iv: "79f1d268532b0c679fd9035b4ff0e6b8"
server_key_version: "encryptionkey@0"
I20260620 00:07:38.032059 10960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:07:38.056728 10960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:07:38.059163 10960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:07:38.059337 10960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:07:38.059587 10960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:07:38.059911 10960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:07:38.059966 10960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:38.060007 10960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:07:38.060052 10960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:07:38.066172 10960 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38587
I20260620 00:07:38.066215 11088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38587 every 8 connection(s)
I20260620 00:07:38.066547 10960 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 00:07:38 dist-test-slave-grqt krb5kdc[10585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914058, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:07:38.075562 10670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:38.067282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39595 (local address 127.0.98.62:41809)
0620 00:07:38.067344 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:38.067346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:38.067498 (+ 152us) server_negotiation.cc:407] Connection header received
0620 00:07:38.067971 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:38.067973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:38.067984 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:38.068050 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:38.068947 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:38.069456 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:38.070413 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:38.070590 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:38.070640 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:38.073072 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:38.073074 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:38.073074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:38.073081 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:38.074393 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:38.074759 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:38.074760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:38.074760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:38.074812 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:38.075034 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:38.075035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:38.075035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:38.075137 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:38.075189 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:38.075421 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 00:07:38.075494 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 00:07:38.075748 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:38.067221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41809 (local address 127.0.98.3:39595)
0620 00:07:38.067374 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:38.067414 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:38.067879 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:38.068106 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:38.068109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:38.068663 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:07:38.068853 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:38.068858 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:38.069551 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:38.069552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:38.070340 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:38.070341 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:38.070659 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:38.070659 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:38.070763 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:38.071325 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:07:38.071333 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:07:38.072985 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:07:38.074464 (+ 1479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:38.074470 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:38.074472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:38.074690 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:38.074872 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:07:38.074873 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:07:38.074873 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:07:38.074980 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:07:38.075191 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:07:38.075195 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:07:38.075309 (+ 114us) client_negotiation.cc:769] Sending connection context
0620 00:07:38.075390 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 00:07:38.075532 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
I20260620 00:07:38.076444 11089 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41809
I20260620 00:07:38.076606 11089 heartbeater.cc:464] Registering TS with master...
I20260620 00:07:38.076610 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 10960
I20260620 00:07:38.076696 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:07:38.076836 11089 heartbeater.cc:511] Master 127.0.98.62:41809 requested a full tablet report, sending...
I20260620 00:07:38.076892 392 external_mini_cluster.cc:1428] Setting key 986253cceca657ae58722a420ede61fb
I20260620 00:07:38.077311 10630 ts_manager.cc:194] Registered new tserver with Master: 58fb7c9f8e3c4924b76c6ae3a40f9ce4 (127.0.98.3:38587)
I20260620 00:07:38.077733 10630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39595
I20260620 00:07:38.077814 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:07:38.085301 10670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:38.079163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57168 (local address 127.0.98.62:41809)
0620 00:07:38.079208 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:38.079210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:38.079304 (+ 94us) server_negotiation.cc:407] Connection header received
0620 00:07:38.079396 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:38.079397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:38.079407 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:38.079473 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:07:38.080007 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:38.080491 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:38.081291 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:38.081651 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:38.081720 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:38.082906 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:07:38.082907 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:07:38.082908 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:07:38.082915 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:07:38.084184 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:38.084576 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:38.084577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:38.084577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:38.084635 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:07:38.084826 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:07:38.084827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:07:38.084828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:07:38.084982 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:07:38.085067 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:38.085172 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 00:07:38.085233 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 00:07:38.086886 10630 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57168:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:07:38.088358 10630 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:07:38.094851 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:38.090973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52018 (local address 127.0.98.1:36921)
0620 00:07:38.091171 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:38.091175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:38.091184 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:07:38.091221 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:38.091223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:38.091237 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:38.091460 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:07:38.091527 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:38.092644 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:38.093922 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:38.094416 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:38.094511 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:38.094571 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:38.094724 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 00:07:38.094760 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":63,"threads_started":1}
I20260620 00:07:38.094852 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:38.090698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36921 (local address 127.0.0.1:52018)
0620 00:07:38.090810 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:38.090851 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:38.090920 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:38.091326 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:38.091327 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:38.091328 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:07:38.091384 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:38.091386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:38.092751 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:38.092752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:38.093833 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:38.093836 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:38.094515 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:38.094516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:38.094595 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:38.094608 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 00:07:38.094707 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 00:07:38.094761 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":32,"thread_start_us":70,"threads_started":1}
I20260620 00:07:38.095016 11102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:38.090698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44117 (local address 127.0.0.1:48088)
0620 00:07:38.090862 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:38.090874 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:38.090940 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:38.091480 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:38.091480 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:38.091483 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:07:38.091531 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:38.091534 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:38.092509 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:38.092510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:38.093865 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:38.093867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:38.094797 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:38.094797 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:38.094857 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:38.094862 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:07:38.094891 (+ 29us) client_negotiation.cc:240] Negotiation successful
0620 00:07:38.094941 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":5,"thread_start_us":91,"threads_started":1}
I20260620 00:07:38.095391 11105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:38.091132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48088 (local address 127.0.98.2:44117)
0620 00:07:38.091292 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:38.091296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:38.091307 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:07:38.091347 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:38.091350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:38.091365 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:38.091615 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:07:38.091672 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:38.092340 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:38.093980 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:38.094708 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:38.095183 (+ 475us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:38.095246 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:38.095270 (+ 24us) server_negotiation.cc:299] Negotiation successful
0620 00:07:38.095309 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
I20260620 00:07:38.095782 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:38.090705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38587 (local address 127.0.0.1:49208)
0620 00:07:38.091021 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:38.091029 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:38.091065 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:38.091298 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:38.091301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:38.091303 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:07:38.091384 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:38.091402 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:38.093624 (+ 2222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:38.093625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:38.094663 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:38.094666 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:38.095602 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:38.095603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:38.095666 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:38.095670 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:07:38.095708 (+ 38us) client_negotiation.cc:240] Negotiation successful
0620 00:07:38.095737 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":2,"thread_start_us":85,"threads_started":1}
I20260620 00:07:38.095980 10889 tablet_service.cc:1467] Processing CreateTablet for tablet 1d48f1ee73e049b4a652635068bb19c3 (DEFAULT_TABLE table=test-table [id=65dcfa1fd3e0440ab92f19d7ed969982]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:07:38.095979 10755 tablet_service.cc:1467] Processing CreateTablet for tablet 1d48f1ee73e049b4a652635068bb19c3 (DEFAULT_TABLE table=test-table [id=65dcfa1fd3e0440ab92f19d7ed969982]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:07:38.096165 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:38.090935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49208 (local address 127.0.98.3:38587)
0620 00:07:38.091108 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:38.091114 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:38.091126 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:07:38.091166 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:38.091168 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:38.091184 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:38.091482 (+ 298us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:07:38.091550 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:38.092639 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:38.094768 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:38.095503 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:38.095901 (+ 398us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:38.095969 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:38.096000 (+ 31us) server_negotiation.cc:299] Negotiation successful
0620 00:07:38.096033 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":89,"threads_started":1}
I20260620 00:07:38.096328 10889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d48f1ee73e049b4a652635068bb19c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:38.096328 10755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d48f1ee73e049b4a652635068bb19c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:38.096732 11023 tablet_service.cc:1467] Processing CreateTablet for tablet 1d48f1ee73e049b4a652635068bb19c3 (DEFAULT_TABLE table=test-table [id=65dcfa1fd3e0440ab92f19d7ed969982]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:07:38.096961 11023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d48f1ee73e049b4a652635068bb19c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:07:38.099877 11107 tablet_bootstrap.cc:492] T 1d48f1ee73e049b4a652635068bb19c3 P a252b24e99b94769b31d9a50d12dc538: Bootstrap starting.
I20260620 00:07:38.100407 11107 tablet_bootstrap.cc:654] T 1d48f1ee73e049b4a652635068bb19c3 P a252b24e99b94769b31d9a50d12dc538: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:38.100844 11107 log.cc:824] T 1d48f1ee73e049b4a652635068bb19c3 P a252b24e99b94769b31d9a50d12dc538: Log is configured to *not* fsync() on all Append() calls
I20260620 00:07:38.101257 11106 tablet_bootstrap.cc:492] T 1d48f1ee73e049b4a652635068bb19c3 P 58fb7c9f8e3c4924b76c6ae3a40f9ce4: Bootstrap starting.
I20260620 00:07:38.101521 11108 tablet_bootstrap.cc:492] T 1d48f1ee73e049b4a652635068bb19c3 P 068a3d41f2ca4bff9d581eac08ac19c4: Bootstrap starting.
I20260620 00:07:38.101868 11106 tablet_bootstrap.cc:654] T 1d48f1ee73e049b4a652635068bb19c3 P 58fb7c9f8e3c4924b76c6ae3a40f9ce4: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:38.102298 11108 tablet_bootstrap.cc:654] T 1d48f1ee73e049b4a652635068bb19c3 P 068a3d41f2ca4bff9d581eac08ac19c4: Neither blocks nor log segments found. Creating new log.
I20260620 00:07:38.102463 11107 tablet_bootstrap.cc:492] T 1d48f1ee73e049b4a652635068bb19c3 P a252b24e99b94769b31d9a50d12dc538: No bootstrap required, opened a new log
I20260620 00:07:38.102517 11107 ts_tablet_manager.cc:1397] T 1d48f1ee73e049b4a652635068bb19c3 P a252b24e99b94769b31d9a50d12dc538: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 00:07:38.102829 11108 log.cc:824] T 1d48f1ee73e049b4a652635068bb19c3 P 068a3d41f2ca4bff9d581eac08ac19c4: Log is configured to *not* fsync() on all Append() calls
I20260620 00:07:38.103535 11108 tablet_bootstrap.cc:492] T 1d48f1ee73e049b4a652635068bb19c3 P 068a3d41f2ca4bff9d581eac08ac19c4: No bootstrap required, opened a new log
I20260620 00:07:38.103617 11108 ts_tablet_manager.cc:1397] T 1d48f1ee73e049b4a652635068bb19c3 P 068a3d41f2ca4bff9d581eac08ac19c4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:07:38.104173 11107 raft_consensus.cc:348] T 1d48f1ee73e049b4a652635068bb19c3 P a252b24e99b94769b31d9a50d12dc538 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a252b24e99b94769b31d9a50d12dc538" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "068a3d41f2ca4bff9d581eac08ac19c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36921 } } peers { permanent_uuid: "58fb7c9f8e3c4924b76c6ae3a40f9ce4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38587 } }
I20260620 00:07:38.104318 11107 raft_consensus.cc:374] T 1d48f1ee73e049b4a652635068bb19c3 P a252b24e99b94769b31d9a50d12dc538 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:38.104341 11107 raft_consensus.cc:729] T 1d48f1ee73e049b4a652635068bb19c3 P a252b24e99b94769b31d9a50d12dc538 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a252b24e99b94769b31d9a50d12dc538, State: Initialized, Role: FOLLOWER
I20260620 00:07:38.104472 11107 consensus_queue.cc:260] T 1d48f1ee73e049b4a652635068bb19c3 P a252b24e99b94769b31d9a50d12dc538 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a252b24e99b94769b31d9a50d12dc538" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "068a3d41f2ca4bff9d581eac08ac19c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36921 } } peers { permanent_uuid: "58fb7c9f8e3c4924b76c6ae3a40f9ce4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38587 } }
I20260620 00:07:38.104774 11107 ts_tablet_manager.cc:1428] T 1d48f1ee73e049b4a652635068bb19c3 P a252b24e99b94769b31d9a50d12dc538: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 00:07:38.104774 10955 heartbeater.cc:503] Master 127.0.98.62:41809 was elected leader, sending a full tablet report...
I20260620 00:07:38.105270 11106 log.cc:824] T 1d48f1ee73e049b4a652635068bb19c3 P 58fb7c9f8e3c4924b76c6ae3a40f9ce4: Log is configured to *not* fsync() on all Append() calls
I20260620 00:07:38.105352 11108 raft_consensus.cc:348] T 1d48f1ee73e049b4a652635068bb19c3 P 068a3d41f2ca4bff9d581eac08ac19c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a252b24e99b94769b31d9a50d12dc538" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "068a3d41f2ca4bff9d581eac08ac19c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36921 } } peers { permanent_uuid: "58fb7c9f8e3c4924b76c6ae3a40f9ce4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38587 } }
I20260620 00:07:38.105623 11108 raft_consensus.cc:374] T 1d48f1ee73e049b4a652635068bb19c3 P 068a3d41f2ca4bff9d581eac08ac19c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:38.105680 11108 raft_consensus.cc:729] T 1d48f1ee73e049b4a652635068bb19c3 P 068a3d41f2ca4bff9d581eac08ac19c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 068a3d41f2ca4bff9d581eac08ac19c4, State: Initialized, Role: FOLLOWER
I20260620 00:07:38.105805 11108 consensus_queue.cc:260] T 1d48f1ee73e049b4a652635068bb19c3 P 068a3d41f2ca4bff9d581eac08ac19c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a252b24e99b94769b31d9a50d12dc538" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "068a3d41f2ca4bff9d581eac08ac19c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36921 } } peers { permanent_uuid: "58fb7c9f8e3c4924b76c6ae3a40f9ce4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38587 } }
I20260620 00:07:38.106068 11106 tablet_bootstrap.cc:492] T 1d48f1ee73e049b4a652635068bb19c3 P 58fb7c9f8e3c4924b76c6ae3a40f9ce4: No bootstrap required, opened a new log
I20260620 00:07:38.106094 11108 ts_tablet_manager.cc:1428] T 1d48f1ee73e049b4a652635068bb19c3 P 068a3d41f2ca4bff9d581eac08ac19c4: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 00:07:38.106132 11106 ts_tablet_manager.cc:1397] T 1d48f1ee73e049b4a652635068bb19c3 P 58fb7c9f8e3c4924b76c6ae3a40f9ce4: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260620 00:07:38.106348 10821 heartbeater.cc:503] Master 127.0.98.62:41809 was elected leader, sending a full tablet report...
I20260620 00:07:38.107759 11106 raft_consensus.cc:348] T 1d48f1ee73e049b4a652635068bb19c3 P 58fb7c9f8e3c4924b76c6ae3a40f9ce4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a252b24e99b94769b31d9a50d12dc538" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "068a3d41f2ca4bff9d581eac08ac19c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36921 } } peers { permanent_uuid: "58fb7c9f8e3c4924b76c6ae3a40f9ce4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38587 } }
I20260620 00:07:38.107882 11106 raft_consensus.cc:374] T 1d48f1ee73e049b4a652635068bb19c3 P 58fb7c9f8e3c4924b76c6ae3a40f9ce4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:07:38.107900 11106 raft_consensus.cc:729] T 1d48f1ee73e049b4a652635068bb19c3 P 58fb7c9f8e3c4924b76c6ae3a40f9ce4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58fb7c9f8e3c4924b76c6ae3a40f9ce4, State: Initialized, Role: FOLLOWER
I20260620 00:07:38.108011 11106 consensus_queue.cc:260] T 1d48f1ee73e049b4a652635068bb19c3 P 58fb7c9f8e3c4924b76c6ae3a40f9ce4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a252b24e99b94769b31d9a50d12dc538" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "068a3d41f2ca4bff9d581eac08ac19c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36921 } } peers { permanent_uuid: "58fb7c9f8e3c4924b76c6ae3a40f9ce4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38587 } }
I20260620 00:07:38.108218 11106 ts_tablet_manager.cc:1428] T 1d48f1ee73e049b4a652635068bb19c3 P 58fb7c9f8e3c4924b76c6ae3a40f9ce4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 00:07:38.108299 11089 heartbeater.cc:503] Master 127.0.98.62:41809 was elected leader, sending a full tablet report...
W20260620 00:07:38.145195 10956 tablet.cc:2369] T 1d48f1ee73e049b4a652635068bb19c3 P a252b24e99b94769b31d9a50d12dc538: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 00:07:38.232303 10822 tablet.cc:2369] T 1d48f1ee73e049b4a652635068bb19c3 P 068a3d41f2ca4bff9d581eac08ac19c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:07:38.278831 11114 raft_consensus.cc:482] T 1d48f1ee73e049b4a652635068bb19c3 P 58fb7c9f8e3c4924b76c6ae3a40f9ce4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:07:38.278970 11114 raft_consensus.cc:504] T 1d48f1ee73e049b4a652635068bb19c3 P 58fb7c9f8e3c4924b76c6ae3a40f9ce4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a252b24e99b94769b31d9a50d12dc538" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "068a3d41f2ca4bff9d581eac08ac19c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36921 } } peers { permanent_uuid: "58fb7c9f8e3c4924b76c6ae3a40f9ce4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38587 } }
I20260620 00:07:38.279553 11114 leader_election.cc:288] T 1d48f1ee73e049b4a652635068bb19c3 P 58fb7c9f8e3c4924b76c6ae3a40f9ce4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a252b24e99b94769b31d9a50d12dc538 (127.0.98.2:44117), 068a3d41f2ca4bff9d581eac08ac19c4 (127.0.98.1:36921)
I20260620 00:07:38.282855 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:38.279788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44117 (local address 127.0.98.3:47105)
0620 00:07:38.279861 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:38.279875 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:38.279946 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:38.280142 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:38.280143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:38.280144 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:07:38.280218 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:38.280220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:38.280965 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:38.280966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:38.281900 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:38.281903 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:38.282569 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:38.282570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:38.282654 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:38.282658 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:07:38.282720 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 00:07:38.282762 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 00:07:38.282869 11105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:38.279882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47105 (local address 127.0.98.2:44117)
0620 00:07:38.279937 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:38.279939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:38.279950 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:07:38.280069 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:38.280071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:38.280084 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:38.280137 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:07:38.280305 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:38.280866 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:38.282020 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:38.282495 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:38.282554 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:38.282564 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:38.282740 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 00:07:38.282774 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 00:07:38.283177 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:38.279941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38237 (local address 127.0.98.1:36921)
0620 00:07:38.280003 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:38.280005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:38.280136 (+ 131us) server_negotiation.cc:407] Connection header received
0620 00:07:38.280233 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:38.280234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:38.280247 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:38.280303 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:07:38.280543 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:38.281082 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:38.282293 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:38.282765 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:38.282829 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:38.282838 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:38.283046 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 00:07:38.283099 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 00:07:38.283313 10909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d48f1ee73e049b4a652635068bb19c3" candidate_uuid: "58fb7c9f8e3c4924b76c6ae3a40f9ce4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a252b24e99b94769b31d9a50d12dc538" is_pre_election: true
I20260620 00:07:38.283424 11115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:38.279798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36921 (local address 127.0.98.3:38237)
0620 00:07:38.280042 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:38.280051 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:07:38.280167 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:07:38.280310 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:07:38.280310 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:07:38.280312 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:07:38.280381 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:38.280382 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:38.281225 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:38.281226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:38.282161 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:07:38.282163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:07:38.282850 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:07:38.282851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:07:38.282934 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:38.282938 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:07:38.283302 (+ 364us) client_negotiation.cc:240] Negotiation successful
0620 00:07:38.283351 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":85,"thread_start_us":95,"threads_started":1}
I20260620 00:07:38.283481 10909 raft_consensus.cc:2454] T 1d48f1ee73e049b4a652635068bb19c3 P a252b24e99b94769b31d9a50d12dc538 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58fb7c9f8e3c4924b76c6ae3a40f9ce4 in term 0.
I20260620 00:07:38.283725 10979 leader_election.cc:302] T 1d48f1ee73e049b4a652635068bb19c3 P 58fb7c9f8e3c4924b76c6ae3a40f9ce4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58fb7c9f8e3c4924b76c6ae3a40f9ce4, a252b24e99b94769b31d9a50d12dc538; no voters:
I20260620 00:07:38.283782 10775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d48f1ee73e049b4a652635068bb19c3" candidate_uuid: "58fb7c9f8e3c4924b76c6ae3a40f9ce4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "068a3d41f2ca4bff9d581eac08ac19c4" is_pre_election: true
I20260620 00:07:38.283874 11114 raft_consensus.cc:2781] T 1d48f1ee73e049b4a652635068bb19c3 P 58fb7c9f8e3c4924b76c6ae3a40f9ce4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:07:38.283923 11114 raft_consensus.cc:482] T 1d48f1ee73e049b4a652635068bb19c3 P 58fb7c9f8e3c4924b76c6ae3a40f9ce4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:07:38.283943 10775 raft_consensus.cc:2454] T 1d48f1ee73e049b4a652635068bb19c3 P 068a3d41f2ca4bff9d581eac08ac19c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58fb7c9f8e3c4924b76c6ae3a40f9ce4 in term 0.
I20260620 00:07:38.283967 11114 raft_consensus.cc:3037] T 1d48f1ee73e049b4a652635068bb19c3 P 58fb7c9f8e3c4924b76c6ae3a40f9ce4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:38.284732 11114 raft_consensus.cc:504] T 1d48f1ee73e049b4a652635068bb19c3 P 58fb7c9f8e3c4924b76c6ae3a40f9ce4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a252b24e99b94769b31d9a50d12dc538" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "068a3d41f2ca4bff9d581eac08ac19c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36921 } } peers { permanent_uuid: "58fb7c9f8e3c4924b76c6ae3a40f9ce4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38587 } }
I20260620 00:07:38.284943 11114 leader_election.cc:288] T 1d48f1ee73e049b4a652635068bb19c3 P 58fb7c9f8e3c4924b76c6ae3a40f9ce4 [CANDIDATE]: Term 1 election: Requested vote from peers a252b24e99b94769b31d9a50d12dc538 (127.0.98.2:44117), 068a3d41f2ca4bff9d581eac08ac19c4 (127.0.98.1:36921)
I20260620 00:07:38.285061 10775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d48f1ee73e049b4a652635068bb19c3" candidate_uuid: "58fb7c9f8e3c4924b76c6ae3a40f9ce4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "068a3d41f2ca4bff9d581eac08ac19c4"
I20260620 00:07:38.285122 10909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d48f1ee73e049b4a652635068bb19c3" candidate_uuid: "58fb7c9f8e3c4924b76c6ae3a40f9ce4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a252b24e99b94769b31d9a50d12dc538"
I20260620 00:07:38.285154 10775 raft_consensus.cc:3037] T 1d48f1ee73e049b4a652635068bb19c3 P 068a3d41f2ca4bff9d581eac08ac19c4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:38.285202 10909 raft_consensus.cc:3037] T 1d48f1ee73e049b4a652635068bb19c3 P a252b24e99b94769b31d9a50d12dc538 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:07:38.285962 10775 raft_consensus.cc:2454] T 1d48f1ee73e049b4a652635068bb19c3 P 068a3d41f2ca4bff9d581eac08ac19c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58fb7c9f8e3c4924b76c6ae3a40f9ce4 in term 1.
I20260620 00:07:38.286031 10909 raft_consensus.cc:2454] T 1d48f1ee73e049b4a652635068bb19c3 P a252b24e99b94769b31d9a50d12dc538 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58fb7c9f8e3c4924b76c6ae3a40f9ce4 in term 1.
I20260620 00:07:38.286194 10976 leader_election.cc:302] T 1d48f1ee73e049b4a652635068bb19c3 P 58fb7c9f8e3c4924b76c6ae3a40f9ce4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 068a3d41f2ca4bff9d581eac08ac19c4, 58fb7c9f8e3c4924b76c6ae3a40f9ce4; no voters:
I20260620 00:07:38.286320 11114 raft_consensus.cc:2781] T 1d48f1ee73e049b4a652635068bb19c3 P 58fb7c9f8e3c4924b76c6ae3a40f9ce4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:07:38.286489 11114 raft_consensus.cc:686] T 1d48f1ee73e049b4a652635068bb19c3 P 58fb7c9f8e3c4924b76c6ae3a40f9ce4 [term 1 LEADER]: Becoming Leader. State: Replica: 58fb7c9f8e3c4924b76c6ae3a40f9ce4, State: Running, Role: LEADER
I20260620 00:07:38.286592 11114 consensus_queue.cc:237] T 1d48f1ee73e049b4a652635068bb19c3 P 58fb7c9f8e3c4924b76c6ae3a40f9ce4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a252b24e99b94769b31d9a50d12dc538" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "068a3d41f2ca4bff9d581eac08ac19c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36921 } } peers { permanent_uuid: "58fb7c9f8e3c4924b76c6ae3a40f9ce4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38587 } }
I20260620 00:07:38.287528 10630 catalog_manager.cc:5495] T 1d48f1ee73e049b4a652635068bb19c3 P 58fb7c9f8e3c4924b76c6ae3a40f9ce4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58fb7c9f8e3c4924b76c6ae3a40f9ce4 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "58fb7c9f8e3c4924b76c6ae3a40f9ce4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a252b24e99b94769b31d9a50d12dc538" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "068a3d41f2ca4bff9d581eac08ac19c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58fb7c9f8e3c4924b76c6ae3a40f9ce4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38587 } health_report { overall_health: HEALTHY } } }
W20260620 00:07:38.317103 11090 tablet.cc:2369] T 1d48f1ee73e049b4a652635068bb19c3 P 58fb7c9f8e3c4924b76c6ae3a40f9ce4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:07:38.330304 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:38.327534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49222 (local address 127.0.98.3:38587)
0620 00:07:38.327604 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 00:07:38.327606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:07:38.327616 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:07:38.327691 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:07:38.327691 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:07:38.327711 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:07:38.327809 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:07:38.328002 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:38.328759 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:38.329625 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:07:38.329802 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:07:38.329856 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:07:38.330018 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:07:38.330113 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:07:38.330175 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 00:07:38.330211 (+ 36us) server_negotiation.cc:299] Negotiation successful
0620 00:07:38.330228 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 00:07:38.332130 10775 raft_consensus.cc:1261] T 1d48f1ee73e049b4a652635068bb19c3 P 068a3d41f2ca4bff9d581eac08ac19c4 [term 1 FOLLOWER]: Refusing update from remote peer 58fb7c9f8e3c4924b76c6ae3a40f9ce4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:07:38.332130 10909 raft_consensus.cc:1261] T 1d48f1ee73e049b4a652635068bb19c3 P a252b24e99b94769b31d9a50d12dc538 [term 1 FOLLOWER]: Refusing update from remote peer 58fb7c9f8e3c4924b76c6ae3a40f9ce4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:07:38.332530 11116 consensus_queue.cc:1035] T 1d48f1ee73e049b4a652635068bb19c3 P 58fb7c9f8e3c4924b76c6ae3a40f9ce4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "068a3d41f2ca4bff9d581eac08ac19c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:07:38.332647 11114 consensus_queue.cc:1035] T 1d48f1ee73e049b4a652635068bb19c3 P 58fb7c9f8e3c4924b76c6ae3a40f9ce4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a252b24e99b94769b31d9a50d12dc538" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:07:38.335067 11120 mvcc.cc:205] Tried to move back new op lower bound from 7298719982925561856 to 7298719982742573056. Current Snapshot: MvccSnapshot[applied={T|T < 7298719982925561856}]
I20260620 00:07:38.336103 11119 mvcc.cc:205] Tried to move back new op lower bound from 7298719982925561856 to 7298719982742573056. Current Snapshot: MvccSnapshot[applied={T|T < 7298719982925561856 or (T in {7298719982925561856})}]
I20260620 00:07:38.337153 10629 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:07:38.337244 10629 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:07:38.338137 10629 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1000dfe1dc264e16aeb9e817801e380d: Sending DeleteTablet for 3 replicas of tablet 1d48f1ee73e049b4a652635068bb19c3
I20260620 00:07:38.338371 10889 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d48f1ee73e049b4a652635068bb19c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:07:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48088
I20260620 00:07:38.338426 10755 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d48f1ee73e049b4a652635068bb19c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:07:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52018
I20260620 00:07:38.338631 11129 tablet_replica.cc:330] T 1d48f1ee73e049b4a652635068bb19c3 P 068a3d41f2ca4bff9d581eac08ac19c4: stopping tablet replica
I20260620 00:07:38.338615 10888 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d48f1ee73e049b4a652635068bb19c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:07:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48088
I20260620 00:07:38.338718 11129 raft_consensus.cc:2229] T 1d48f1ee73e049b4a652635068bb19c3 P 068a3d41f2ca4bff9d581eac08ac19c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:07:38.338722 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 10692
I20260620 00:07:38.338789 11128 tablet_replica.cc:330] T 1d48f1ee73e049b4a652635068bb19c3 P a252b24e99b94769b31d9a50d12dc538: stopping tablet replica
I20260620 00:07:38.338853 11128 raft_consensus.cc:2229] T 1d48f1ee73e049b4a652635068bb19c3 P a252b24e99b94769b31d9a50d12dc538 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:07:38.338922 11128 raft_consensus.cc:2258] T 1d48f1ee73e049b4a652635068bb19c3 P a252b24e99b94769b31d9a50d12dc538 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:07:38.339706 11128 ts_tablet_manager.cc:1905] T 1d48f1ee73e049b4a652635068bb19c3 P a252b24e99b94769b31d9a50d12dc538: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:07:38.341519 11128 ts_tablet_manager.cc:1918] T 1d48f1ee73e049b4a652635068bb19c3 P a252b24e99b94769b31d9a50d12dc538: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:07:38.341596 11128 log.cc:1196] T 1d48f1ee73e049b4a652635068bb19c3 P a252b24e99b94769b31d9a50d12dc538: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914001714546-392-0/minicluster-data/ts-1/wal/wals/1d48f1ee73e049b4a652635068bb19c3
I20260620 00:07:38.341876 11128 ts_tablet_manager.cc:1939] T 1d48f1ee73e049b4a652635068bb19c3 P a252b24e99b94769b31d9a50d12dc538: Deleting consensus metadata
I20260620 00:07:38.342356 10617 catalog_manager.cc:4841] TS a252b24e99b94769b31d9a50d12dc538 (127.0.98.2:44117): tablet 1d48f1ee73e049b4a652635068bb19c3 (table test-table [id=65dcfa1fd3e0440ab92f19d7ed969982]) successfully deleted
W20260620 00:07:38.342504 10617 catalog_manager.cc:4808] TS a252b24e99b94769b31d9a50d12dc538 (127.0.98.2:44117): delete failed for tablet 1d48f1ee73e049b4a652635068bb19c3 because the tablet was not found. No further retry: Not found: Tablet not found: 1d48f1ee73e049b4a652635068bb19c3
W20260620 00:07:38.345167 10614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36921 (error 108)
I20260620 00:07:38.345506 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 10826
I20260620 00:07:38.345479 11023 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d48f1ee73e049b4a652635068bb19c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:07:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49208
I20260620 00:07:38.346100 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:38.345644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36921 (local address 127.0.0.1:52030)
0620 00:07:38.346013 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:38.346062 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346}
W20260620 00:07:38.346225 10614 catalog_manager.cc:4584] TS 068a3d41f2ca4bff9d581eac08ac19c4 (127.0.98.1:36921): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1d48f1ee73e049b4a652635068bb19c3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36921: connect: Connection refused (error 111)
I20260620 00:07:38.350589 11131 tablet_replica.cc:330] T 1d48f1ee73e049b4a652635068bb19c3 P 58fb7c9f8e3c4924b76c6ae3a40f9ce4: stopping tablet replica
I20260620 00:07:38.350741 11131 raft_consensus.cc:2229] T 1d48f1ee73e049b4a652635068bb19c3 P 58fb7c9f8e3c4924b76c6ae3a40f9ce4 [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:07:38.350756 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 10960
I20260620 00:07:38.350870 11131 raft_consensus.cc:2258] T 1d48f1ee73e049b4a652635068bb19c3 P 58fb7c9f8e3c4924b76c6ae3a40f9ce4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:07:38.355620 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 10598
I20260620 00:07:38.355764 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:07:38.355623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38587 (local address 127.0.0.1:49226)
0620 00:07:38.355689 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 00:07:38.355719 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:38587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
2026-06-20T00:07:38Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2193 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:07:38 dist-test-slave-grqt krb5kdc[11134](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:07:38 dist-test-slave-grqt krb5kdc[11134](info): set up 2 sockets
Jun 20 00:07:38 dist-test-slave-grqt krb5kdc[11134](info): commencing operation
krb5kdc: starting...
W20260620 00:07:39.410499 392 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:07:39 dist-test-slave-grqt krb5kdc[11134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:07:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:07:39Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260620 00:07:39.593210 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task8WXHPi/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/postgres -l logfile start
2026-06-20 00:07:42.277 UTC [11164] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 00:07:42.277 UTC [11164] LOG: listening on IPv4 address "127.0.98.20", port 32861
2026-06-20 00:07:42.280 UTC [11164] LOG: listening on Unix socket "/tmp/.s.PGSQL.32861"
2026-06-20 00:07:42.303 UTC [11167] LOG: database system was shut down at 2026-06-20 00:07:40 UTC
2026-06-20 00:07:42.307 UTC [11164] LOG: database system is ready to accept connections
I20260620 00:07:43.254443 392 mini_postgres.cc:96] Postgres bound to 32861
2026-06-20 00:07:43.260 UTC [11175] FATAL: database "slave" does not exist
127.0.98.20:32861 - accepting connections
I20260620 00:07:43.261263 392 mini_ranger.cc:161] Starting Ranger...
I20260620 00:07:43.276882 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 00:07:43.873461 392 mini_ranger.cc:87] Created ranger Postgres database
I20260620 00:07:43.873550 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ranger-admin
2026-06-20 00:07:44,153 [I] DB FLAVOR :POSTGRES
2026-06-20 00:07:44,153 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 00:07:44,153 [I] Checking connection..
2026-06-20 00:07:44,154 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 00:07:44,412 [I] Checking connection passed.
2026-06-20 00:07:44,413 [I] --------- Verifying version history table ---------
2026-06-20 00:07:44,413 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 00:07:44,666 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 00:07:44,666 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 00:07:44,940 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 00:07:44,941 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 00:07:44,941 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32861/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 00:07:45.178 UTC [11239] WARNING: there is no transaction in progress
2026-06-20 00:07:45,185 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 00:07:45,186 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 00:07:45,461 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 00:07:45,461 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 00:07:45,461 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 00:07:45,714 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 00:07:45,985 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-grqt.c.gcp-upstream.internal','N') ;"
2026-06-20 00:07:46,235 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 00:07:46,474 [I] Table x_portal_user does not exist in database ranger
2026-06-20 00:07:46,474 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 00:07:46,732 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 00:07:46,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 00:07:46,969 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 00:07:47,216 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 00:07:47,217 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32861/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 00:07:48.289 UTC [11347] WARNING: there is no transaction in progress
2026-06-20 00:07:48.308 UTC [11347] WARNING: there is no transaction in progress
2026-06-20 00:07:48.327 UTC [11347] WARNING: there is no transaction in progress
2026-06-20 00:07:48.345 UTC [11347] WARNING: there is no transaction in progress
2026-06-20 00:07:48.365 UTC [11347] WARNING: there is no transaction in progress
2026-06-20 00:07:48.386 UTC [11347] WARNING: there is no transaction in progress
2026-06-20 00:07:48.452 UTC [11347] WARNING: there is no transaction in progress
2026-06-20 00:07:48.460 UTC [11347] WARNING: there is no transaction in progress
2026-06-20 00:07:48.477 UTC [11347] WARNING: there is no transaction in progress
2026-06-20 00:07:48.489 UTC [11347] WARNING: there is no transaction in progress
2026-06-20 00:07:48.500 UTC [11347] WARNING: there is no transaction in progress
2026-06-20 00:07:48.516 UTC [11347] WARNING: there is no transaction in progress
2026-06-20 00:07:48.528 UTC [11347] WARNING: there is no transaction in progress
2026-06-20 00:07:48.536 UTC [11347] WARNING: there is no transaction in progress
2026-06-20 00:07:48.544 UTC [11347] WARNING: there is no transaction in progress
2026-06-20 00:07:48.552 UTC [11347] WARNING: there is no transaction in progress
2026-06-20 00:07:49,308 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 00:07:49,309 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 00:07:49,559 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 00:07:49,559 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 00:07:49,807 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 00:07:49,808 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 00:07:50,048 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 00:07:50,048 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 00:07:50,288 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 00:07:50,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-grqt.c.gcp-upstream.internal';"
2026-06-20 00:07:50,782 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 00:07:50,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 00:07:51,037 [I] DB_PATCHES have already been applied
I20260620 00:07:51.042562 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task8WXHPi/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 00:07:51.042661 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260620 00:07:51.045634 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:40751
Jun 20, 2026 12:07:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 12:07:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 12:07:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 12:07:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 12:07:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 12:07:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 12:07:51 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 00:07:52.142089 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.881s user 0.000s sys 0.009s
Jun 20, 2026 12:08:11 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 12:08:11 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 12:08:11 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 12:08:12 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 12:08:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 12:08:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 12:08:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 12:08:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 12:08:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 12:08:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 12:08:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 12:08:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 12:08:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 12:08:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 12:08:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 12:08:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 12:08:12 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 12:08:12 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 00:08:13.487428 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 00:08:13 dist-test-slave-grqt krb5kdc[11134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914093, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 00:08:13.572707 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 00:08:13.583343 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 00:08:14.501466 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 00:08:14.501535 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ranger-kms
I20260620 00:08:14.501549 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:32861
2026-06-20 00:08:14,680 [I] DB FLAVOR :POSTGRES
2026-06-20 00:08:14,680 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 00:08:14,680 [I] Checking connection
2026-06-20 00:08:14,680 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32861/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 00:08:14,946 [I] connection success
2026-06-20 00:08:14,947 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 00:08:14,947 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 00:08:14,947 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32861/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 00:08:15,197 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 00:08:15,198 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 00:08:15,198 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 00:08:15,198 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32861/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 00:08:15,466 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 00:08:15.994187 392 mini_ranger_kms.cc:310] Created kms service
I20260620 00:08:16.199338 392 mini_ranger_kms.cc:326] Created kudu user
I20260620 00:08:16.246639 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 00:08:16 dist-test-slave-grqt krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914093, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:08:16.691850 392 mini_ranger_kms.cc:384] Added ranger policy
I20260620 00:08:16.691969 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 00:08:16.691993 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260620 00:08:16.695238 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11527
I20260620 00:08:16.695370 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:49273
Jun 20, 2026 12:08:17 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 12:08:17 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 12:08:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 12:08:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 12:08:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 12:08:18 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 00:08:18.980697 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.408s user 0.004s sys 0.158s
I20260620 00:08:18.980824 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 00:08:18.980923 392 mini_ranger_kms.cc:401] 127.0.98.20:49273/kms/v1/keys
Jun 20 00:08:19 dist-test-slave-grqt krb5kdc[11134](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781914099, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 00:08:19 dist-test-slave-grqt krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914099, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 00:08:19 dist-test-slave-grqt krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914099, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 00:08:22 dist-test-slave-grqt krb5kdc[11134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:08:22.389815 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40639
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40639
--ranger_config_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:49273/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:08:22.474201 11578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:08:22.474440 11578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:08:22.474464 11578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:08:22.475735 11578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:08:22.475787 11578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:08:22.475807 11578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:08:22.475824 11578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:08:22.475840 11578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:08:22.477589 11578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:49273/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40639
--ranger_config_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:08:22.477840 11578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:08:22.478094 11578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 00:08:22.483603 11578 server_base.cc:1034] running on GCE node
W20260620 00:08:22.483644 11584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:08:22.483753 11583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:08:22.483949 11586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:08:22.484217 11578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:08:22.484560 11578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:08:22.485718 11578 hybrid_clock.cc:648] HybridClock initialized: now 1781914102485698 us; error 38 us; skew 500 ppm
Jun 20 00:08:22 dist-test-slave-grqt krb5kdc[11134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:08:22.493664 11578 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:08:22.494372 11578 webserver.cc:460] Webserver started at http://127.0.98.62:36507/ using document root <none> and password file <none>
I20260620 00:08:22.494625 11578 fs_manager.cc:359] Metadata directory not provided
I20260620 00:08:22.494692 11578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:08:22.494889 11578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 00:08:22 dist-test-slave-grqt krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:08:22.555166 11578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "fc4969b6749a4729abd20e740472f6ee"
format_stamp: "Formatted at 2026-06-20 00:08:22 on dist-test-slave-grqt"
server_key: "2746d9de04ceed7ba3b3baa4f1b444fa"
server_key_iv: "3f56cf067fdd6a08aa1ab46a638abf60"
server_key_version: "kuduclusterkey@0"
I20260620 00:08:22.555647 11578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "fc4969b6749a4729abd20e740472f6ee"
format_stamp: "Formatted at 2026-06-20 00:08:22 on dist-test-slave-grqt"
server_key: "2746d9de04ceed7ba3b3baa4f1b444fa"
server_key_iv: "3f56cf067fdd6a08aa1ab46a638abf60"
server_key_version: "kuduclusterkey@0"
I20260620 00:08:22.557132 11578 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:08:22.569168 11593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:08:22.569429 11578 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:08:22.569531 11578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "fc4969b6749a4729abd20e740472f6ee"
format_stamp: "Formatted at 2026-06-20 00:08:22 on dist-test-slave-grqt"
server_key: "2746d9de04ceed7ba3b3baa4f1b444fa"
server_key_iv: "3f56cf067fdd6a08aa1ab46a638abf60"
server_key_version: "kuduclusterkey@0"
I20260620 00:08:22.569656 11578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:08:22.591084 11578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:08:22.593125 11578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:08:22.593281 11578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:08:22.597939 11578 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40639
I20260620 00:08:22.597965 11645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40639 every 8 connection(s)
I20260620 00:08:22.598340 11578 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:08:22.599134 11646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:08:22.602301 11646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc4969b6749a4729abd20e740472f6ee: Bootstrap starting.
I20260620 00:08:22.603017 11646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc4969b6749a4729abd20e740472f6ee: Neither blocks nor log segments found. Creating new log.
I20260620 00:08:22.603307 11646 log.cc:824] T 00000000000000000000000000000000 P fc4969b6749a4729abd20e740472f6ee: Log is configured to *not* fsync() on all Append() calls
I20260620 00:08:22.604107 11646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc4969b6749a4729abd20e740472f6ee: No bootstrap required, opened a new log
I20260620 00:08:22.606515 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 11578
I20260620 00:08:22.606630 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:08:22.606812 392 external_mini_cluster.cc:1428] Setting key 0d6cf3f42ee4c7518999908edb9e6ed0
I20260620 00:08:22.606877 11646 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc4969b6749a4729abd20e740472f6ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4969b6749a4729abd20e740472f6ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40639 } }
I20260620 00:08:22.607537 11646 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc4969b6749a4729abd20e740472f6ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:08:22.607595 11646 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc4969b6749a4729abd20e740472f6ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc4969b6749a4729abd20e740472f6ee, State: Initialized, Role: FOLLOWER
I20260620 00:08:22.607728 11646 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc4969b6749a4729abd20e740472f6ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4969b6749a4729abd20e740472f6ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40639 } }
I20260620 00:08:22.607832 11646 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc4969b6749a4729abd20e740472f6ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:08:22.607878 11646 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc4969b6749a4729abd20e740472f6ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:08:22.607941 11646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc4969b6749a4729abd20e740472f6ee [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:08:22.608659 11646 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc4969b6749a4729abd20e740472f6ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4969b6749a4729abd20e740472f6ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40639 } }
I20260620 00:08:22.608765 11646 leader_election.cc:302] T 00000000000000000000000000000000 P fc4969b6749a4729abd20e740472f6ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc4969b6749a4729abd20e740472f6ee; no voters:
I20260620 00:08:22.608971 11646 leader_election.cc:288] T 00000000000000000000000000000000 P fc4969b6749a4729abd20e740472f6ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:08:22.609025 11651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc4969b6749a4729abd20e740472f6ee [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:08:22.609194 11651 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc4969b6749a4729abd20e740472f6ee [term 1 LEADER]: Becoming Leader. State: Replica: fc4969b6749a4729abd20e740472f6ee, State: Running, Role: LEADER
I20260620 00:08:22.609299 11646 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc4969b6749a4729abd20e740472f6ee [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:08:22.609344 11651 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc4969b6749a4729abd20e740472f6ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4969b6749a4729abd20e740472f6ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40639 } }
I20260620 00:08:22.609704 11651 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc4969b6749a4729abd20e740472f6ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc4969b6749a4729abd20e740472f6ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4969b6749a4729abd20e740472f6ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40639 } } }
I20260620 00:08:22.609802 11651 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc4969b6749a4729abd20e740472f6ee [sys.catalog]: This master's current role is: LEADER
I20260620 00:08:22.609926 11651 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc4969b6749a4729abd20e740472f6ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc4969b6749a4729abd20e740472f6ee. Latest consensus state: current_term: 1 leader_uuid: "fc4969b6749a4729abd20e740472f6ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4969b6749a4729abd20e740472f6ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40639 } } }
I20260620 00:08:22.609995 11651 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc4969b6749a4729abd20e740472f6ee [sys.catalog]: This master's current role is: LEADER
I20260620 00:08:22.610913 11646 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 00:08:22.615283 11656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:08:22.616186 11656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 00:08:22 dist-test-slave-grqt krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:08:22.622474 11650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:08:22.607353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35272 (local address 127.0.98.62:40639)
0620 00:08:22.607633 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0620 00:08:22.607649 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 00:08:22.607686 (+ 37us) server_negotiation.cc:407] Connection header received
0620 00:08:22.607809 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:08:22.607816 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:08:22.607877 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:08:22.608040 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:08:22.608628 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:08:22.610011 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:08:22.610953 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:08:22.614003 (+ 3050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:08:22.615080 (+ 1077us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:08:22.617299 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:08:22.617304 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:08:22.617305 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:08:22.617322 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:08:22.619922 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:08:22.620402 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:08:22.620406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:08:22.620407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:08:22.620523 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:08:22.620760 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:08:22.620761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:08:22.620761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:08:22.621129 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:08:22.621757 (+ 628us) server_negotiation.cc:1035] Waiting for connection context
0620 00:08:22.621829 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 00:08:22.621964 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":110,"threads_started":1}
I20260620 00:08:22.623869 11656 catalog_manager.cc:1347] Generated new cluster ID: 932a5697fc284d2992db16d0e60fd318
I20260620 00:08:22.624003 11656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:08:22.635725 11656 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:08:22.636621 11656 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:08:22.643981 11656 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc4969b6749a4729abd20e740472f6ee: Generated new TSK 0
I20260620 00:08:22.644299 11656 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:08:23.418171 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:49273/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40639
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:08:23.514493 11691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:08:23.514737 11691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:08:23.514775 11691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:08:23.516062 11691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:08:23.516132 11691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:08:23.516178 11691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:08:23.517854 11691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:49273/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:08:23.518131 11691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:08:23.518386 11691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:08:23.523765 11699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:08:23.523772 11697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:08:23.523845 11696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:08:23.524295 11691 server_base.cc:1034] running on GCE node
I20260620 00:08:23.524549 11691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:08:23.524794 11691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:08:23.525966 11691 hybrid_clock.cc:648] HybridClock initialized: now 1781914103525950 us; error 30 us; skew 500 ppm
Jun 20 00:08:23 dist-test-slave-grqt krb5kdc[11134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:08:23.529094 11691 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:08:23.529687 11691 webserver.cc:460] Webserver started at http://127.0.98.1:44763/ using document root <none> and password file <none>
I20260620 00:08:23.529964 11691 fs_manager.cc:359] Metadata directory not provided
I20260620 00:08:23.530074 11691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:08:23.530221 11691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 00:08:23 dist-test-slave-grqt krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:08:23.545819 11691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-0/data/instance:
uuid: "ea51564306fd4bbcb7c3ddc2f6505feb"
format_stamp: "Formatted at 2026-06-20 00:08:23 on dist-test-slave-grqt"
server_key: "f757a63489804251f20ca68e206fea73"
server_key_iv: "4ad2f05a7ed553ace2891f8a2d346ecd"
server_key_version: "kuduclusterkey@0"
I20260620 00:08:23.546440 11691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ea51564306fd4bbcb7c3ddc2f6505feb"
format_stamp: "Formatted at 2026-06-20 00:08:23 on dist-test-slave-grqt"
server_key: "f757a63489804251f20ca68e206fea73"
server_key_iv: "4ad2f05a7ed553ace2891f8a2d346ecd"
server_key_version: "kuduclusterkey@0"
I20260620 00:08:23.550076 11691 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 00:08:23.564266 11707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:08:23.564495 11691 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:08:23.564594 11691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-0/wal
uuid: "ea51564306fd4bbcb7c3ddc2f6505feb"
format_stamp: "Formatted at 2026-06-20 00:08:23 on dist-test-slave-grqt"
server_key: "f757a63489804251f20ca68e206fea73"
server_key_iv: "4ad2f05a7ed553ace2891f8a2d346ecd"
server_key_version: "kuduclusterkey@0"
I20260620 00:08:23.564716 11691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:08:23.593811 11691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:08:23.596443 11691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:08:23.596612 11691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:08:23.596887 11691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:08:23.597289 11691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:08:23.597412 11691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:08:23.597517 11691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:08:23.597590 11691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:08:23.605094 11691 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43705
I20260620 00:08:23.605401 11821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43705 every 8 connection(s)
I20260620 00:08:23.605741 11691 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:08:23.607039 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 11691
I20260620 00:08:23.607134 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:08:23.607326 392 external_mini_cluster.cc:1428] Setting key dd7d8c1ea3aa687bd8268ca40a45c059
Jun 20 00:08:23 dist-test-slave-grqt krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:08:23.622496 11824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:08:23.606549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40639 (local address 127.0.98.1:36831)
0620 00:08:23.607582 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0620 00:08:23.607634 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 00:08:23.609991 (+ 2357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:08:23.610423 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:08:23.610428 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:08:23.610691 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:08:23.611014 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:08:23.611021 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:08:23.612055 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:08:23.612056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:08:23.613380 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:08:23.613383 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:08:23.613985 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:08:23.613986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:08:23.614127 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:08:23.614766 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:08:23.614777 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:08:23.616557 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:08:23.620476 (+ 3919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:08:23.620488 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:08:23.620491 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:08:23.620891 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:08:23.621187 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:08:23.621187 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:08:23.621188 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:08:23.621340 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:08:23.621703 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:08:23.621711 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:08:23.621924 (+ 213us) client_negotiation.cc:769] Sending connection context
0620 00:08:23.622040 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 00:08:23.622224 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":81,"threads_started":1}
I20260620 00:08:23.623302 11825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:08:23.606584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36831 (local address 127.0.98.62:40639)
0620 00:08:23.610196 (+ 3612us) server_negotiation.cc:206] Beginning negotiation
0620 00:08:23.610198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:08:23.610207 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:08:23.610230 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:08:23.610231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:08:23.610245 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:08:23.611117 (+ 872us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:08:23.611160 (+ 43us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:08:23.611940 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:08:23.613515 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:08:23.613843 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:08:23.613904 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:08:23.616677 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:08:23.616678 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:08:23.616679 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:08:23.616687 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:08:23.620360 (+ 3673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:08:23.621037 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:08:23.621039 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:08:23.621039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:08:23.621113 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:08:23.621454 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:08:23.621455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:08:23.621455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:08:23.621608 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:08:23.621674 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 00:08:23.623112 (+ 1438us) server_negotiation.cc:299] Negotiation successful
0620 00:08:23.623208 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3579,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 00:08:23.629118 11822 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40639
I20260620 00:08:23.629246 11822 heartbeater.cc:464] Registering TS with master...
I20260620 00:08:23.629522 11822 heartbeater.cc:511] Master 127.0.98.62:40639 requested a full tablet report, sending...
I20260620 00:08:23.630170 11610 ts_manager.cc:194] Registered new tserver with Master: ea51564306fd4bbcb7c3ddc2f6505feb (127.0.98.1:43705)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 00:08:23.631110 11610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36831
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:08:23.653905 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:49273/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40639
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:08:23.792237 11828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:08:23.792608 11828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:08:23.792692 11828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:08:23.794682 11828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:08:23.794806 11828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:08:23.794902 11828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
Jun 20 00:08:23 dist-test-slave-grqt krb5kdc[11134](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781914103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:08:23.797870 11828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:49273/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:08:23.798604 11828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:08:23.798933 11828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:08:23.805507 11836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:08:23.806238 11835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:08:23.806860 11828 server_base.cc:1034] running on GCE node
W20260620 00:08:23.807102 11838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:08:23.807392 11828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:08:23.807672 11828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:08:23.808816 11828 hybrid_clock.cc:648] HybridClock initialized: now 1781914103808796 us; error 34 us; skew 500 ppm
Jun 20 00:08:23 dist-test-slave-grqt krb5kdc[11134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:08:23.812556 11828 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:08:23.813077 11828 webserver.cc:460] Webserver started at http://127.0.98.2:34211/ using document root <none> and password file <none>
I20260620 00:08:23.813357 11828 fs_manager.cc:359] Metadata directory not provided
I20260620 00:08:23.813443 11828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:08:23.813575 11828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 00:08:23 dist-test-slave-grqt krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:08:23.827139 11828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-1/data/instance:
uuid: "44a360d8a8a148ebb87a86c8913fc42a"
format_stamp: "Formatted at 2026-06-20 00:08:23 on dist-test-slave-grqt"
server_key: "df142cf66901bd88be1f3dde74fd8d50"
server_key_iv: "0eb5e8c868948ea10b5ba16040f98202"
server_key_version: "kuduclusterkey@0"
I20260620 00:08:23.827695 11828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-1/wal/instance:
uuid: "44a360d8a8a148ebb87a86c8913fc42a"
format_stamp: "Formatted at 2026-06-20 00:08:23 on dist-test-slave-grqt"
server_key: "df142cf66901bd88be1f3dde74fd8d50"
server_key_iv: "0eb5e8c868948ea10b5ba16040f98202"
server_key_version: "kuduclusterkey@0"
I20260620 00:08:23.837733 11828 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.000s sys 0.001s
I20260620 00:08:23.862422 11845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:08:23.862641 11828 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.000s sys 0.001s
I20260620 00:08:23.862733 11828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-1/wal
uuid: "44a360d8a8a148ebb87a86c8913fc42a"
format_stamp: "Formatted at 2026-06-20 00:08:23 on dist-test-slave-grqt"
server_key: "df142cf66901bd88be1f3dde74fd8d50"
server_key_iv: "0eb5e8c868948ea10b5ba16040f98202"
server_key_version: "kuduclusterkey@0"
I20260620 00:08:23.862838 11828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:08:23.874320 11828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:08:23.876729 11828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:08:23.876888 11828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:08:23.877156 11828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:08:23.877562 11828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:08:23.877612 11828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:08:23.877697 11828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:08:23.877727 11828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:08:23.889207 11828 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34633
I20260620 00:08:23.889971 11828 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:08:23.899482 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 11828
I20260620 00:08:23.899590 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:08:23.899783 392 external_mini_cluster.cc:1428] Setting key f53e06dc432b97a2943517f45ed7a77a
I20260620 00:08:23.905786 11958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34633 every 8 connection(s)
Jun 20 00:08:23 dist-test-slave-grqt krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 00:08:23.933280 11825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:08:23.906769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51009 (local address 127.0.98.62:40639)
0620 00:08:23.906828 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 00:08:23.906831 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:08:23.908932 (+ 2101us) server_negotiation.cc:407] Connection header received
0620 00:08:23.910433 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:08:23.910434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:08:23.910448 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:08:23.910479 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:08:23.913539 (+ 3060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:08:23.914301 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:08:23.917462 (+ 3161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:08:23.917787 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:08:23.917849 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:08:23.929506 (+ 11657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:08:23.929508 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:08:23.929509 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:08:23.929520 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:08:23.931254 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:08:23.931981 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:08:23.931983 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:08:23.931983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:08:23.932047 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:08:23.932315 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:08:23.932316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:08:23.932316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:08:23.932457 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:08:23.932506 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 00:08:23.933075 (+ 569us) server_negotiation.cc:299] Negotiation successful
0620 00:08:23.933178 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 00:08:23.936450 11963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:08:23.906491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40639 (local address 127.0.98.2:51009)
0620 00:08:23.908768 (+ 2277us) negotiation.cc:107] Waiting for socket to connect
0620 00:08:23.908838 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 00:08:23.909583 (+ 745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:08:23.910700 (+ 1117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:08:23.910705 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:08:23.911007 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:08:23.911281 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:08:23.911287 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:08:23.914427 (+ 3140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:08:23.914428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:08:23.915683 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:08:23.915686 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:08:23.917953 (+ 2267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:08:23.917954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:08:23.918112 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:08:23.925805 (+ 7693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:08:23.925821 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:08:23.927588 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:08:23.931375 (+ 3787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:08:23.931387 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:08:23.931390 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:08:23.931880 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:08:23.932112 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:08:23.932113 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:08:23.932114 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:08:23.932249 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:08:23.932540 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:08:23.932547 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:08:23.932765 (+ 218us) client_negotiation.cc:769] Sending connection context
0620 00:08:23.933729 (+ 964us) client_negotiation.cc:240] Negotiation successful
0620 00:08:23.934261 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2222,"thread_start_us":90,"threads_started":1}
I20260620 00:08:23.937152 11959 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40639
I20260620 00:08:23.937263 11959 heartbeater.cc:464] Registering TS with master...
I20260620 00:08:23.937546 11959 heartbeater.cc:511] Master 127.0.98.62:40639 requested a full tablet report, sending...
I20260620 00:08:23.937981 11610 ts_manager.cc:194] Registered new tserver with Master: 44a360d8a8a148ebb87a86c8913fc42a (127.0.98.2:34633)
I20260620 00:08:23.938452 11610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51009
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:08:23.954597 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:49273/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40639
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:08:24.111287 11965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:08:24.111665 11965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:08:24.111761 11965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:08:24.113277 11965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:08:24.113356 11965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:08:24.113546 11965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:08:24.115765 11965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:49273/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:08:24.116302 11965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:08:24.116639 11965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:08:24.123111 11973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:08:24.123111 11971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:08:24.123209 11970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:08:24.124435 11965 server_base.cc:1034] running on GCE node
I20260620 00:08:24.124812 11965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:08:24.125131 11965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:08:24.132519 11965 hybrid_clock.cc:648] HybridClock initialized: now 1781914104132473 us; error 57 us; skew 500 ppm
Jun 20 00:08:24 dist-test-slave-grqt krb5kdc[11134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:08:24.136284 11965 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:08:24.136835 11965 webserver.cc:460] Webserver started at http://127.0.98.3:36799/ using document root <none> and password file <none>
I20260620 00:08:24.137079 11965 fs_manager.cc:359] Metadata directory not provided
I20260620 00:08:24.137126 11965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:08:24.137248 11965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 00:08:24 dist-test-slave-grqt krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:08:24.152069 11965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-2/data/instance:
uuid: "99d106fe0e8245ef9adad8faf43193ff"
format_stamp: "Formatted at 2026-06-20 00:08:24 on dist-test-slave-grqt"
server_key: "336efee94568bbc662faefafa236c1c9"
server_key_iv: "47e92a54cc30ef9bf9bbbff1dadf2505"
server_key_version: "kuduclusterkey@0"
I20260620 00:08:24.152652 11965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-2/wal/instance:
uuid: "99d106fe0e8245ef9adad8faf43193ff"
format_stamp: "Formatted at 2026-06-20 00:08:24 on dist-test-slave-grqt"
server_key: "336efee94568bbc662faefafa236c1c9"
server_key_iv: "47e92a54cc30ef9bf9bbbff1dadf2505"
server_key_version: "kuduclusterkey@0"
I20260620 00:08:24.156106 11965 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260620 00:08:24.170008 11980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:08:24.170262 11965 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 00:08:24.170351 11965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-2/wal
uuid: "99d106fe0e8245ef9adad8faf43193ff"
format_stamp: "Formatted at 2026-06-20 00:08:24 on dist-test-slave-grqt"
server_key: "336efee94568bbc662faefafa236c1c9"
server_key_iv: "47e92a54cc30ef9bf9bbbff1dadf2505"
server_key_version: "kuduclusterkey@0"
I20260620 00:08:24.170455 11965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:08:24.189594 11965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:08:24.192040 11965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:08:24.192256 11965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:08:24.192565 11965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:08:24.192963 11965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:08:24.193053 11965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:08:24.193127 11965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:08:24.193182 11965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:08:24.199934 11965 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33681
I20260620 00:08:24.200469 11965 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:08:24.205245 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 11965
I20260620 00:08:24.205346 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:08:24.205574 392 external_mini_cluster.cc:1428] Setting key 1944d4c36f4291ec48d0c585881cebe3
I20260620 00:08:24.203696 12093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33681 every 8 connection(s)
Jun 20 00:08:24 dist-test-slave-grqt krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:08:24.219349 11825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:08:24.207170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35701 (local address 127.0.98.62:40639)
0620 00:08:24.207212 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 00:08:24.207214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:08:24.208045 (+ 831us) server_negotiation.cc:407] Connection header received
0620 00:08:24.208765 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:08:24.208766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:08:24.208778 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:08:24.208835 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:08:24.209673 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:08:24.210538 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:08:24.212071 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:08:24.212398 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:08:24.212443 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:08:24.215260 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:08:24.215261 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:08:24.215261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:08:24.215269 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:08:24.217032 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:08:24.218162 (+ 1130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:08:24.218164 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:08:24.218164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:08:24.218227 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:08:24.218547 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:08:24.218548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:08:24.218548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:08:24.218691 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:08:24.218744 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 00:08:24.219147 (+ 403us) server_negotiation.cc:299] Negotiation successful
0620 00:08:24.219246 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 00:08:24.220162 12096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:08:24.207194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40639 (local address 127.0.98.3:35701)
0620 00:08:24.207896 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0620 00:08:24.207958 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 00:08:24.208628 (+ 670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:08:24.208925 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:08:24.208930 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:08:24.209188 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:08:24.209558 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:08:24.209564 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:08:24.210659 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:08:24.210660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:08:24.211965 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:08:24.211968 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:08:24.212543 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:08:24.212543 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:08:24.212692 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:08:24.213346 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:08:24.213356 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:08:24.215136 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:08:24.217733 (+ 2597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:08:24.217742 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:08:24.217745 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:08:24.218070 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:08:24.218303 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:08:24.218304 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:08:24.218304 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:08:24.218465 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:08:24.218778 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:08:24.218785 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:08:24.218982 (+ 197us) client_negotiation.cc:769] Sending connection context
0620 00:08:24.219742 (+ 760us) client_negotiation.cc:240] Negotiation successful
0620 00:08:24.219919 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":109,"threads_started":1}
I20260620 00:08:24.220842 12094 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40639
I20260620 00:08:24.220990 12094 heartbeater.cc:464] Registering TS with master...
I20260620 00:08:24.221226 12094 heartbeater.cc:511] Master 127.0.98.62:40639 requested a full tablet report, sending...
I20260620 00:08:24.221639 11610 ts_manager.cc:194] Registered new tserver with Master: 99d106fe0e8245ef9adad8faf43193ff (127.0.98.3:33681)
I20260620 00:08:24.222126 11610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35701
I20260620 00:08:24.228499 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:08:24.237942 11825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:08:24.230417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35284 (local address 127.0.98.62:40639)
0620 00:08:24.230466 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 00:08:24.230469 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:08:24.230648 (+ 179us) server_negotiation.cc:407] Connection header received
0620 00:08:24.230749 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:08:24.230750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:08:24.230761 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:08:24.230791 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:08:24.231532 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:08:24.232190 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:08:24.232938 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:08:24.233146 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:08:24.233240 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:08:24.234911 (+ 1671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:08:24.234913 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:08:24.234913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:08:24.234921 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:08:24.236637 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:08:24.237048 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:08:24.237050 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:08:24.237050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:08:24.237110 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:08:24.237311 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:08:24.237312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:08:24.237312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:08:24.237469 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:08:24.237549 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 00:08:24.237786 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 00:08:24.237871 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 00:08:24.240000 11610 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35284:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:08:24.241647 11610 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:08:24.252427 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:08:24.246159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59400 (local address 127.0.98.1:43705)
0620 00:08:24.246311 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 00:08:24.246317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:08:24.246667 (+ 350us) server_negotiation.cc:407] Connection header received
0620 00:08:24.246811 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:08:24.246813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:08:24.246831 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:08:24.246899 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:08:24.247163 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:08:24.248923 (+ 1760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:08:24.251111 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:08:24.251894 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:08:24.251946 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:08:24.252008 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 00:08:24.252289 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 00:08:24.252333 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":66,"threads_started":1}
I20260620 00:08:24.253252 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:08:24.245887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43705 (local address 127.0.0.1:59400)
0620 00:08:24.246544 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0620 00:08:24.246590 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 00:08:24.246708 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:08:24.247010 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:08:24.247013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:08:24.247015 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:08:24.247083 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:08:24.247085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:08:24.249026 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:08:24.249027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:08:24.250994 (+ 1967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:08:24.250997 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:08:24.252080 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:08:24.252081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:08:24.252161 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:08:24.252175 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 00:08:24.252781 (+ 606us) client_negotiation.cc:240] Negotiation successful
0620 00:08:24.253171 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":93,"threads_started":1}
I20260620 00:08:24.254328 11742 tablet_service.cc:1467] Processing CreateTablet for tablet 2b74359ecaca407a8b060cf6091b191d (DEFAULT_TABLE table=test-table [id=dc5a91a9264040f5bc9684d968a628fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:08:24.254724 11742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b74359ecaca407a8b060cf6091b191d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:08:24.260313 12109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:08:24.246383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34633 (local address 127.0.0.1:56354)
0620 00:08:24.247474 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0620 00:08:24.247484 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:08:24.247526 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:08:24.247945 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:08:24.247946 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:08:24.247948 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:08:24.247996 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:08:24.247997 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:08:24.249639 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:08:24.249640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:08:24.250923 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:08:24.250927 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:08:24.260014 (+ 9087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:08:24.260015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:08:24.260105 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:08:24.260111 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 00:08:24.260160 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 00:08:24.260216 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1074,"thread_start_us":84,"threads_started":1}
I20260620 00:08:24.260888 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:08:24.246894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56354 (local address 127.0.98.2:34633)
0620 00:08:24.247639 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0620 00:08:24.247644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:08:24.247652 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:08:24.247688 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:08:24.247690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:08:24.247704 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:08:24.247786 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:08:24.248077 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:08:24.249556 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:08:24.251726 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:08:24.252488 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:08:24.252559 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:08:24.252628 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 00:08:24.260704 (+ 8076us) server_negotiation.cc:299] Negotiation successful
0620 00:08:24.260785 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":80,"threads_started":1}
I20260620 00:08:24.261251 12112 tablet_bootstrap.cc:492] T 2b74359ecaca407a8b060cf6091b191d P ea51564306fd4bbcb7c3ddc2f6505feb: Bootstrap starting.
I20260620 00:08:24.261956 11893 tablet_service.cc:1467] Processing CreateTablet for tablet 2b74359ecaca407a8b060cf6091b191d (DEFAULT_TABLE table=test-table [id=dc5a91a9264040f5bc9684d968a628fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:08:24.262328 11893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b74359ecaca407a8b060cf6091b191d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:08:24.263235 12112 tablet_bootstrap.cc:654] T 2b74359ecaca407a8b060cf6091b191d P ea51564306fd4bbcb7c3ddc2f6505feb: Neither blocks nor log segments found. Creating new log.
I20260620 00:08:24.263522 12112 log.cc:824] T 2b74359ecaca407a8b060cf6091b191d P ea51564306fd4bbcb7c3ddc2f6505feb: Log is configured to *not* fsync() on all Append() calls
I20260620 00:08:24.265550 12112 tablet_bootstrap.cc:492] T 2b74359ecaca407a8b060cf6091b191d P ea51564306fd4bbcb7c3ddc2f6505feb: No bootstrap required, opened a new log
I20260620 00:08:24.265620 12112 ts_tablet_manager.cc:1397] T 2b74359ecaca407a8b060cf6091b191d P ea51564306fd4bbcb7c3ddc2f6505feb: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 00:08:24.267777 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:08:24.246559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33681 (local address 127.0.0.1:53430)
0620 00:08:24.251062 (+ 4503us) negotiation.cc:107] Waiting for socket to connect
0620 00:08:24.251073 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:08:24.251118 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:08:24.252663 (+ 1545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:08:24.252664 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:08:24.252666 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:08:24.252708 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:08:24.252710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:08:24.260495 (+ 7785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:08:24.260496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:08:24.264825 (+ 4329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:08:24.264830 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:08:24.267519 (+ 2689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:08:24.267521 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:08:24.267599 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:08:24.267606 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 00:08:24.267641 (+ 35us) client_negotiation.cc:240] Negotiation successful
0620 00:08:24.267694 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4482,"thread_start_us":55,"threads_started":1}
I20260620 00:08:24.267864 12112 raft_consensus.cc:348] T 2b74359ecaca407a8b060cf6091b191d P ea51564306fd4bbcb7c3ddc2f6505feb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea51564306fd4bbcb7c3ddc2f6505feb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43705 } } peers { permanent_uuid: "44a360d8a8a148ebb87a86c8913fc42a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34633 } } peers { permanent_uuid: "99d106fe0e8245ef9adad8faf43193ff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33681 } }
I20260620 00:08:24.268057 12112 raft_consensus.cc:374] T 2b74359ecaca407a8b060cf6091b191d P ea51564306fd4bbcb7c3ddc2f6505feb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:08:24.268088 12112 raft_consensus.cc:729] T 2b74359ecaca407a8b060cf6091b191d P ea51564306fd4bbcb7c3ddc2f6505feb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea51564306fd4bbcb7c3ddc2f6505feb, State: Initialized, Role: FOLLOWER
I20260620 00:08:24.268209 12112 consensus_queue.cc:260] T 2b74359ecaca407a8b060cf6091b191d P ea51564306fd4bbcb7c3ddc2f6505feb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea51564306fd4bbcb7c3ddc2f6505feb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43705 } } peers { permanent_uuid: "44a360d8a8a148ebb87a86c8913fc42a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34633 } } peers { permanent_uuid: "99d106fe0e8245ef9adad8faf43193ff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33681 } }
I20260620 00:08:24.268246 12111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:08:24.247058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53430 (local address 127.0.98.3:33681)
0620 00:08:24.247911 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0620 00:08:24.247915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:08:24.251180 (+ 3265us) server_negotiation.cc:407] Connection header received
0620 00:08:24.251227 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:08:24.251230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:08:24.251247 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:08:24.251319 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:08:24.252847 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:08:24.259908 (+ 7061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:08:24.266448 (+ 6540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:08:24.267398 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:08:24.267961 (+ 563us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:08:24.268059 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 00:08:24.268089 (+ 30us) server_negotiation.cc:299] Negotiation successful
0620 00:08:24.268133 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":819,"thread_start_us":71,"threads_started":1}
I20260620 00:08:24.268465 12112 ts_tablet_manager.cc:1428] T 2b74359ecaca407a8b060cf6091b191d P ea51564306fd4bbcb7c3ddc2f6505feb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:08:24.269003 11822 heartbeater.cc:503] Master 127.0.98.62:40639 was elected leader, sending a full tablet report...
I20260620 00:08:24.268910 12028 tablet_service.cc:1467] Processing CreateTablet for tablet 2b74359ecaca407a8b060cf6091b191d (DEFAULT_TABLE table=test-table [id=dc5a91a9264040f5bc9684d968a628fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:08:24.269284 12028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b74359ecaca407a8b060cf6091b191d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:08:24.269656 12114 tablet_bootstrap.cc:492] T 2b74359ecaca407a8b060cf6091b191d P 44a360d8a8a148ebb87a86c8913fc42a: Bootstrap starting.
I20260620 00:08:24.270780 12114 tablet_bootstrap.cc:654] T 2b74359ecaca407a8b060cf6091b191d P 44a360d8a8a148ebb87a86c8913fc42a: Neither blocks nor log segments found. Creating new log.
I20260620 00:08:24.271157 12114 log.cc:824] T 2b74359ecaca407a8b060cf6091b191d P 44a360d8a8a148ebb87a86c8913fc42a: Log is configured to *not* fsync() on all Append() calls
I20260620 00:08:24.271935 12114 tablet_bootstrap.cc:492] T 2b74359ecaca407a8b060cf6091b191d P 44a360d8a8a148ebb87a86c8913fc42a: No bootstrap required, opened a new log
I20260620 00:08:24.271999 12114 ts_tablet_manager.cc:1397] T 2b74359ecaca407a8b060cf6091b191d P 44a360d8a8a148ebb87a86c8913fc42a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:08:24.273952 12114 raft_consensus.cc:348] T 2b74359ecaca407a8b060cf6091b191d P 44a360d8a8a148ebb87a86c8913fc42a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea51564306fd4bbcb7c3ddc2f6505feb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43705 } } peers { permanent_uuid: "44a360d8a8a148ebb87a86c8913fc42a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34633 } } peers { permanent_uuid: "99d106fe0e8245ef9adad8faf43193ff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33681 } }
I20260620 00:08:24.274115 12114 raft_consensus.cc:374] T 2b74359ecaca407a8b060cf6091b191d P 44a360d8a8a148ebb87a86c8913fc42a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:08:24.274145 12114 raft_consensus.cc:729] T 2b74359ecaca407a8b060cf6091b191d P 44a360d8a8a148ebb87a86c8913fc42a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44a360d8a8a148ebb87a86c8913fc42a, State: Initialized, Role: FOLLOWER
I20260620 00:08:24.274255 12114 consensus_queue.cc:260] T 2b74359ecaca407a8b060cf6091b191d P 44a360d8a8a148ebb87a86c8913fc42a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea51564306fd4bbcb7c3ddc2f6505feb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43705 } } peers { permanent_uuid: "44a360d8a8a148ebb87a86c8913fc42a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34633 } } peers { permanent_uuid: "99d106fe0e8245ef9adad8faf43193ff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33681 } }
I20260620 00:08:24.274492 12114 ts_tablet_manager.cc:1428] T 2b74359ecaca407a8b060cf6091b191d P 44a360d8a8a148ebb87a86c8913fc42a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:08:24.274622 11959 heartbeater.cc:503] Master 127.0.98.62:40639 was elected leader, sending a full tablet report...
I20260620 00:08:24.275753 12118 tablet_bootstrap.cc:492] T 2b74359ecaca407a8b060cf6091b191d P 99d106fe0e8245ef9adad8faf43193ff: Bootstrap starting.
I20260620 00:08:24.276855 12118 tablet_bootstrap.cc:654] T 2b74359ecaca407a8b060cf6091b191d P 99d106fe0e8245ef9adad8faf43193ff: Neither blocks nor log segments found. Creating new log.
I20260620 00:08:24.277222 12118 log.cc:824] T 2b74359ecaca407a8b060cf6091b191d P 99d106fe0e8245ef9adad8faf43193ff: Log is configured to *not* fsync() on all Append() calls
I20260620 00:08:24.278098 12118 tablet_bootstrap.cc:492] T 2b74359ecaca407a8b060cf6091b191d P 99d106fe0e8245ef9adad8faf43193ff: No bootstrap required, opened a new log
I20260620 00:08:24.278175 12118 ts_tablet_manager.cc:1397] T 2b74359ecaca407a8b060cf6091b191d P 99d106fe0e8245ef9adad8faf43193ff: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:08:24.280246 12118 raft_consensus.cc:348] T 2b74359ecaca407a8b060cf6091b191d P 99d106fe0e8245ef9adad8faf43193ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea51564306fd4bbcb7c3ddc2f6505feb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43705 } } peers { permanent_uuid: "44a360d8a8a148ebb87a86c8913fc42a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34633 } } peers { permanent_uuid: "99d106fe0e8245ef9adad8faf43193ff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33681 } }
I20260620 00:08:24.280426 12118 raft_consensus.cc:374] T 2b74359ecaca407a8b060cf6091b191d P 99d106fe0e8245ef9adad8faf43193ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:08:24.280463 12118 raft_consensus.cc:729] T 2b74359ecaca407a8b060cf6091b191d P 99d106fe0e8245ef9adad8faf43193ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99d106fe0e8245ef9adad8faf43193ff, State: Initialized, Role: FOLLOWER
I20260620 00:08:24.280577 12118 consensus_queue.cc:260] T 2b74359ecaca407a8b060cf6091b191d P 99d106fe0e8245ef9adad8faf43193ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea51564306fd4bbcb7c3ddc2f6505feb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43705 } } peers { permanent_uuid: "44a360d8a8a148ebb87a86c8913fc42a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34633 } } peers { permanent_uuid: "99d106fe0e8245ef9adad8faf43193ff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33681 } }
I20260620 00:08:24.280867 12118 ts_tablet_manager.cc:1428] T 2b74359ecaca407a8b060cf6091b191d P 99d106fe0e8245ef9adad8faf43193ff: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:08:24.281446 12094 heartbeater.cc:503] Master 127.0.98.62:40639 was elected leader, sending a full tablet report...
I20260620 00:08:24.281853 12115 raft_consensus.cc:482] T 2b74359ecaca407a8b060cf6091b191d P ea51564306fd4bbcb7c3ddc2f6505feb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:08:24.281985 12115 raft_consensus.cc:504] T 2b74359ecaca407a8b060cf6091b191d P ea51564306fd4bbcb7c3ddc2f6505feb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea51564306fd4bbcb7c3ddc2f6505feb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43705 } } peers { permanent_uuid: "44a360d8a8a148ebb87a86c8913fc42a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34633 } } peers { permanent_uuid: "99d106fe0e8245ef9adad8faf43193ff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33681 } }
I20260620 00:08:24.282295 12115 leader_election.cc:288] T 2b74359ecaca407a8b060cf6091b191d P ea51564306fd4bbcb7c3ddc2f6505feb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44a360d8a8a148ebb87a86c8913fc42a (127.0.98.2:34633), 99d106fe0e8245ef9adad8faf43193ff (127.0.98.3:33681)
I20260620 00:08:24.286907 12111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:08:24.283177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33053 (local address 127.0.98.3:33681)
0620 00:08:24.283352 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 00:08:24.283353 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:08:24.283360 (+ 7us) server_negotiation.cc:407] Connection header received
0620 00:08:24.283381 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:08:24.283382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:08:24.283390 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:08:24.283441 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:08:24.283637 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:08:24.284369 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:08:24.285951 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:08:24.286474 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:08:24.286524 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:08:24.286536 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 00:08:24.286768 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 00:08:24.286811 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260620 00:08:24.287176 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:08:24.282689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33681 (local address 127.0.98.1:33053)
0620 00:08:24.282888 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 00:08:24.282898 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:08:24.282938 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:08:24.283471 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:08:24.283473 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:08:24.283475 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:08:24.283550 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:08:24.283552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:08:24.284489 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:08:24.284490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:08:24.285842 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:08:24.285846 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:08:24.286616 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:08:24.286617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:08:24.286688 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:08:24.286692 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:08:24.287033 (+ 341us) client_negotiation.cc:240] Negotiation successful
0620 00:08:24.287077 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":75,"threads_started":1}
I20260620 00:08:24.287449 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:08:24.283129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59061 (local address 127.0.98.2:34633)
0620 00:08:24.283197 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 00:08:24.283199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:08:24.283208 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:08:24.283233 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:08:24.283234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:08:24.283245 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:08:24.283296 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:08:24.283645 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:08:24.284386 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:08:24.286282 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:08:24.286794 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:08:24.286839 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:08:24.286848 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 00:08:24.287345 (+ 497us) server_negotiation.cc:299] Negotiation successful
0620 00:08:24.287385 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 00:08:24.287559 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b74359ecaca407a8b060cf6091b191d" candidate_uuid: "ea51564306fd4bbcb7c3ddc2f6505feb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d106fe0e8245ef9adad8faf43193ff" is_pre_election: true
I20260620 00:08:24.287739 12048 raft_consensus.cc:2454] T 2b74359ecaca407a8b060cf6091b191d P 99d106fe0e8245ef9adad8faf43193ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea51564306fd4bbcb7c3ddc2f6505feb in term 0.
I20260620 00:08:24.287989 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:08:24.282481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34633 (local address 127.0.98.1:59061)
0620 00:08:24.282674 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 00:08:24.282690 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 00:08:24.282769 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:08:24.283486 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:08:24.283487 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:08:24.283489 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:08:24.283564 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:08:24.283566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:08:24.284506 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:08:24.284507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:08:24.286175 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:08:24.286178 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:08:24.286940 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:08:24.286941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:08:24.287197 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:08:24.287201 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:08:24.287822 (+ 621us) client_negotiation.cc:240] Negotiation successful
0620 00:08:24.287874 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":103,"threads_started":1}
I20260620 00:08:24.287986 11710 leader_election.cc:302] T 2b74359ecaca407a8b060cf6091b191d P ea51564306fd4bbcb7c3ddc2f6505feb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99d106fe0e8245ef9adad8faf43193ff, ea51564306fd4bbcb7c3ddc2f6505feb; no voters:
I20260620 00:08:24.288427 12115 raft_consensus.cc:2781] T 2b74359ecaca407a8b060cf6091b191d P ea51564306fd4bbcb7c3ddc2f6505feb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:08:24.288496 12115 raft_consensus.cc:482] T 2b74359ecaca407a8b060cf6091b191d P ea51564306fd4bbcb7c3ddc2f6505feb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:08:24.288522 12115 raft_consensus.cc:3037] T 2b74359ecaca407a8b060cf6091b191d P ea51564306fd4bbcb7c3ddc2f6505feb [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:08:24.288570 11913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b74359ecaca407a8b060cf6091b191d" candidate_uuid: "ea51564306fd4bbcb7c3ddc2f6505feb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44a360d8a8a148ebb87a86c8913fc42a" is_pre_election: true
I20260620 00:08:24.288699 11913 raft_consensus.cc:2454] T 2b74359ecaca407a8b060cf6091b191d P 44a360d8a8a148ebb87a86c8913fc42a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea51564306fd4bbcb7c3ddc2f6505feb in term 0.
I20260620 00:08:24.289903 12115 raft_consensus.cc:504] T 2b74359ecaca407a8b060cf6091b191d P ea51564306fd4bbcb7c3ddc2f6505feb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea51564306fd4bbcb7c3ddc2f6505feb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43705 } } peers { permanent_uuid: "44a360d8a8a148ebb87a86c8913fc42a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34633 } } peers { permanent_uuid: "99d106fe0e8245ef9adad8faf43193ff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33681 } }
I20260620 00:08:24.290053 12115 leader_election.cc:288] T 2b74359ecaca407a8b060cf6091b191d P ea51564306fd4bbcb7c3ddc2f6505feb [CANDIDATE]: Term 1 election: Requested vote from peers 44a360d8a8a148ebb87a86c8913fc42a (127.0.98.2:34633), 99d106fe0e8245ef9adad8faf43193ff (127.0.98.3:33681)
I20260620 00:08:24.290243 11913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b74359ecaca407a8b060cf6091b191d" candidate_uuid: "ea51564306fd4bbcb7c3ddc2f6505feb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44a360d8a8a148ebb87a86c8913fc42a"
I20260620 00:08:24.290282 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b74359ecaca407a8b060cf6091b191d" candidate_uuid: "ea51564306fd4bbcb7c3ddc2f6505feb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d106fe0e8245ef9adad8faf43193ff"
I20260620 00:08:24.290313 11913 raft_consensus.cc:3037] T 2b74359ecaca407a8b060cf6091b191d P 44a360d8a8a148ebb87a86c8913fc42a [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:08:24.290356 12048 raft_consensus.cc:3037] T 2b74359ecaca407a8b060cf6091b191d P 99d106fe0e8245ef9adad8faf43193ff [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:08:24.291131 11913 raft_consensus.cc:2454] T 2b74359ecaca407a8b060cf6091b191d P 44a360d8a8a148ebb87a86c8913fc42a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea51564306fd4bbcb7c3ddc2f6505feb in term 1.
I20260620 00:08:24.291256 12048 raft_consensus.cc:2454] T 2b74359ecaca407a8b060cf6091b191d P 99d106fe0e8245ef9adad8faf43193ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea51564306fd4bbcb7c3ddc2f6505feb in term 1.
I20260620 00:08:24.291466 11710 leader_election.cc:302] T 2b74359ecaca407a8b060cf6091b191d P ea51564306fd4bbcb7c3ddc2f6505feb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99d106fe0e8245ef9adad8faf43193ff, ea51564306fd4bbcb7c3ddc2f6505feb; no voters:
I20260620 00:08:24.291616 12115 raft_consensus.cc:2781] T 2b74359ecaca407a8b060cf6091b191d P ea51564306fd4bbcb7c3ddc2f6505feb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:08:24.291792 12115 raft_consensus.cc:686] T 2b74359ecaca407a8b060cf6091b191d P ea51564306fd4bbcb7c3ddc2f6505feb [term 1 LEADER]: Becoming Leader. State: Replica: ea51564306fd4bbcb7c3ddc2f6505feb, State: Running, Role: LEADER
I20260620 00:08:24.291900 12115 consensus_queue.cc:237] T 2b74359ecaca407a8b060cf6091b191d P ea51564306fd4bbcb7c3ddc2f6505feb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea51564306fd4bbcb7c3ddc2f6505feb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43705 } } peers { permanent_uuid: "44a360d8a8a148ebb87a86c8913fc42a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34633 } } peers { permanent_uuid: "99d106fe0e8245ef9adad8faf43193ff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33681 } }
I20260620 00:08:24.292793 11610 catalog_manager.cc:5495] T 2b74359ecaca407a8b060cf6091b191d P ea51564306fd4bbcb7c3ddc2f6505feb reported cstate change: term changed from 0 to 1, leader changed from <none> to ea51564306fd4bbcb7c3ddc2f6505feb (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ea51564306fd4bbcb7c3ddc2f6505feb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea51564306fd4bbcb7c3ddc2f6505feb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44a360d8a8a148ebb87a86c8913fc42a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99d106fe0e8245ef9adad8faf43193ff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33681 } health_report { overall_health: UNKNOWN } } }
I20260620 00:08:24.304811 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:08:24.301531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59416 (local address 127.0.98.1:43705)
0620 00:08:24.301584 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 00:08:24.301586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:08:24.301597 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:08:24.301680 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:08:24.301681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:08:24.301703 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:08:24.301781 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:08:24.301941 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:08:24.302679 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:08:24.303705 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:08:24.303993 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:08:24.304046 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:08:24.304253 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:08:24.304391 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:08:24.304469 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 00:08:24.304709 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 00:08:24.304731 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 00:08:24.307161 12048 raft_consensus.cc:1261] T 2b74359ecaca407a8b060cf6091b191d P 99d106fe0e8245ef9adad8faf43193ff [term 1 FOLLOWER]: Refusing update from remote peer ea51564306fd4bbcb7c3ddc2f6505feb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:08:24.307188 11913 raft_consensus.cc:1261] T 2b74359ecaca407a8b060cf6091b191d P 44a360d8a8a148ebb87a86c8913fc42a [term 1 FOLLOWER]: Refusing update from remote peer ea51564306fd4bbcb7c3ddc2f6505feb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:08:24.307548 12115 consensus_queue.cc:1035] T 2b74359ecaca407a8b060cf6091b191d P ea51564306fd4bbcb7c3ddc2f6505feb [LEADER]: Connected to new peer: Peer: permanent_uuid: "44a360d8a8a148ebb87a86c8913fc42a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:08:24.307672 12123 consensus_queue.cc:1035] T 2b74359ecaca407a8b060cf6091b191d P ea51564306fd4bbcb7c3ddc2f6505feb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d106fe0e8245ef9adad8faf43193ff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:08:24.312814 12126 mvcc.cc:205] Tried to move back new op lower bound from 7298720171237990400 to 7298720171180023808. Current Snapshot: MvccSnapshot[applied={T|T < 7298720171237990400}]
I20260620 00:08:24.314275 12127 mvcc.cc:205] Tried to move back new op lower bound from 7298720171237990400 to 7298720171180023808. Current Snapshot: MvccSnapshot[applied={T|T < 7298720171237990400}]
I20260620 00:08:24.314460 12129 mvcc.cc:205] Tried to move back new op lower bound from 7298720171237990400 to 7298720171180023808. Current Snapshot: MvccSnapshot[applied={T|T < 7298720171237990400}]
I20260620 00:08:24.317519 11610 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:08:24.317634 11610 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:08:24.318419 11610 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc4969b6749a4729abd20e740472f6ee: Sending DeleteTablet for 3 replicas of tablet 2b74359ecaca407a8b060cf6091b191d
I20260620 00:08:24.318775 11742 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b74359ecaca407a8b060cf6091b191d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:08:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59400
I20260620 00:08:24.318965 11893 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b74359ecaca407a8b060cf6091b191d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:08:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56354
I20260620 00:08:24.319077 12136 tablet_replica.cc:330] T 2b74359ecaca407a8b060cf6091b191d P ea51564306fd4bbcb7c3ddc2f6505feb: stopping tablet replica
I20260620 00:08:24.319207 12028 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b74359ecaca407a8b060cf6091b191d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:08:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53430
I20260620 00:08:24.319808 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 11691
I20260620 00:08:24.320452 12137 tablet_replica.cc:330] T 2b74359ecaca407a8b060cf6091b191d P 44a360d8a8a148ebb87a86c8913fc42a: stopping tablet replica
I20260620 00:08:24.320555 12137 raft_consensus.cc:2229] T 2b74359ecaca407a8b060cf6091b191d P 44a360d8a8a148ebb87a86c8913fc42a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:08:24.320655 12137 raft_consensus.cc:2258] T 2b74359ecaca407a8b060cf6091b191d P 44a360d8a8a148ebb87a86c8913fc42a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:08:24.320830 12138 tablet_replica.cc:330] T 2b74359ecaca407a8b060cf6091b191d P 99d106fe0e8245ef9adad8faf43193ff: stopping tablet replica
I20260620 00:08:24.320937 12138 raft_consensus.cc:2229] T 2b74359ecaca407a8b060cf6091b191d P 99d106fe0e8245ef9adad8faf43193ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:08:24.321033 12138 raft_consensus.cc:2258] T 2b74359ecaca407a8b060cf6091b191d P 99d106fe0e8245ef9adad8faf43193ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:08:24.321632 12137 ts_tablet_manager.cc:1905] T 2b74359ecaca407a8b060cf6091b191d P 44a360d8a8a148ebb87a86c8913fc42a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:08:24.323341 12137 ts_tablet_manager.cc:1918] T 2b74359ecaca407a8b060cf6091b191d P 44a360d8a8a148ebb87a86c8913fc42a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:08:24.323417 12137 log.cc:1196] T 2b74359ecaca407a8b060cf6091b191d P 44a360d8a8a148ebb87a86c8913fc42a: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-1/wal/wals/2b74359ecaca407a8b060cf6091b191d
I20260620 00:08:24.324028 12137 ts_tablet_manager.cc:1939] T 2b74359ecaca407a8b060cf6091b191d P 44a360d8a8a148ebb87a86c8913fc42a: Deleting consensus metadata
I20260620 00:08:24.324494 12138 ts_tablet_manager.cc:1905] T 2b74359ecaca407a8b060cf6091b191d P 99d106fe0e8245ef9adad8faf43193ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:08:24.326629 11595 catalog_manager.cc:4841] TS 44a360d8a8a148ebb87a86c8913fc42a (127.0.98.2:34633): tablet 2b74359ecaca407a8b060cf6091b191d (table test-table [id=dc5a91a9264040f5bc9684d968a628fb]) successfully deleted
I20260620 00:08:24.326709 12138 ts_tablet_manager.cc:1918] T 2b74359ecaca407a8b060cf6091b191d P 99d106fe0e8245ef9adad8faf43193ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:08:24.326781 12138 log.cc:1196] T 2b74359ecaca407a8b060cf6091b191d P 99d106fe0e8245ef9adad8faf43193ff: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914001714546-392-0/minicluster-data/ts-2/wal/wals/2b74359ecaca407a8b060cf6091b191d
I20260620 00:08:24.327482 12138 ts_tablet_manager.cc:1939] T 2b74359ecaca407a8b060cf6091b191d P 99d106fe0e8245ef9adad8faf43193ff: Deleting consensus metadata
I20260620 00:08:24.327956 11595 catalog_manager.cc:4841] TS 99d106fe0e8245ef9adad8faf43193ff (127.0.98.3:33681): tablet 2b74359ecaca407a8b060cf6091b191d (table test-table [id=dc5a91a9264040f5bc9684d968a628fb]) successfully deleted
W20260620 00:08:24.328295 11596 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43705 (error 108)
I20260620 00:08:24.328588 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 11828
I20260620 00:08:24.334452 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:08:24.334287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43705 (local address 127.0.0.1:59418)
0620 00:08:24.334359 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0620 00:08:24.334406 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 00:08:24.334915 11596 catalog_manager.cc:4584] TS ea51564306fd4bbcb7c3ddc2f6505feb (127.0.98.1:43705): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2b74359ecaca407a8b060cf6091b191d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43705: connect: Connection refused (error 111)
I20260620 00:08:24.335083 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 11965
I20260620 00:08:24.340580 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 11578
I20260620 00:08:24.349640 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 00:08:24.687845 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 00:08:24.687955 392 mini_ranger.cc:66] Stopping Ranger...
I20260620 00:08:25.033179 392 mini_ranger.cc:68] Stopped Ranger
2026-06-20 00:08:25.033 UTC [11164] LOG: received smart shutdown request
2026-06-20 00:08:25.036 UTC [11164] LOG: background worker "logical replication launcher" (PID 11173) exited with exit code 1
2026-06-20 00:08:25.037 UTC [11168] LOG: shutting down
2026-06-20 00:08:25.200 UTC [11164] LOG: database system is shut down
2026-06-20T00:08:25Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46953 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:08:25 dist-test-slave-grqt krb5kdc[12148](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:08:25 dist-test-slave-grqt krb5kdc[12148](info): set up 2 sockets
Jun 20 00:08:25 dist-test-slave-grqt krb5kdc[12148](info): commencing operation
krb5kdc: starting...
W20260620 00:08:26.365568 392 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:08:26 dist-test-slave-grqt krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:08:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:08:26Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260620 00:08:26.553679 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task8WXHPi/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/postgres -l logfile start
2026-06-20 00:08:29.485 UTC [12178] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 00:08:29.485 UTC [12178] LOG: listening on IPv4 address "127.0.98.20", port 41643
2026-06-20 00:08:29.491 UTC [12178] LOG: listening on Unix socket "/tmp/.s.PGSQL.41643"
2026-06-20 00:08:29.513 UTC [12181] LOG: database system was shut down at 2026-06-20 00:08:27 UTC
2026-06-20 00:08:29.518 UTC [12178] LOG: database system is ready to accept connections
I20260620 00:08:30.485412 392 mini_postgres.cc:96] Postgres bound to 41643
2026-06-20 00:08:30.491 UTC [12189] FATAL: database "slave" does not exist
127.0.98.20:41643 - accepting connections
I20260620 00:08:30.492574 392 mini_ranger.cc:161] Starting Ranger...
I20260620 00:08:30.508386 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 00:08:31.064361 392 mini_ranger.cc:87] Created ranger Postgres database
I20260620 00:08:31.064456 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ranger-admin
2026-06-20 00:08:31,364 [I] DB FLAVOR :POSTGRES
2026-06-20 00:08:31,365 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 00:08:31,365 [I] Checking connection..
2026-06-20 00:08:31,365 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 00:08:31,619 [I] Checking connection passed.
2026-06-20 00:08:31,620 [I] --------- Verifying version history table ---------
2026-06-20 00:08:31,620 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 00:08:31,895 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 00:08:31,896 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 00:08:32,149 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 00:08:32,150 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 00:08:32,150 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41643/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 00:08:32.415 UTC [12253] WARNING: there is no transaction in progress
2026-06-20 00:08:32,422 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 00:08:32,423 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 00:08:32,673 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 00:08:32,673 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 00:08:32,674 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 00:08:32,931 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 00:08:33,168 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-grqt.c.gcp-upstream.internal','N') ;"
2026-06-20 00:08:33,426 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 00:08:33,696 [I] Table x_portal_user does not exist in database ranger
2026-06-20 00:08:33,696 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 00:08:33,967 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 00:08:33,968 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 00:08:34,254 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 00:08:34,507 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 00:08:34,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41643/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 00:08:35.502 UTC [12361] WARNING: there is no transaction in progress
2026-06-20 00:08:35.520 UTC [12361] WARNING: there is no transaction in progress
2026-06-20 00:08:35.537 UTC [12361] WARNING: there is no transaction in progress
2026-06-20 00:08:35.554 UTC [12361] WARNING: there is no transaction in progress
2026-06-20 00:08:35.571 UTC [12361] WARNING: there is no transaction in progress
2026-06-20 00:08:35.589 UTC [12361] WARNING: there is no transaction in progress
2026-06-20 00:08:35.648 UTC [12361] WARNING: there is no transaction in progress
2026-06-20 00:08:35.656 UTC [12361] WARNING: there is no transaction in progress
2026-06-20 00:08:35.671 UTC [12361] WARNING: there is no transaction in progress
2026-06-20 00:08:35.683 UTC [12361] WARNING: there is no transaction in progress
2026-06-20 00:08:35.694 UTC [12361] WARNING: there is no transaction in progress
2026-06-20 00:08:35.709 UTC [12361] WARNING: there is no transaction in progress
2026-06-20 00:08:35.720 UTC [12361] WARNING: there is no transaction in progress
2026-06-20 00:08:35.727 UTC [12361] WARNING: there is no transaction in progress
2026-06-20 00:08:35.733 UTC [12361] WARNING: there is no transaction in progress
2026-06-20 00:08:35.741 UTC [12361] WARNING: there is no transaction in progress
2026-06-20 00:08:36,523 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 00:08:36,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 00:08:36,808 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 00:08:36,808 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 00:08:37,072 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 00:08:37,072 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 00:08:37,328 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 00:08:37,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 00:08:37,580 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 00:08:37,832 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-grqt.c.gcp-upstream.internal';"
2026-06-20 00:08:38,108 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 00:08:38,108 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 00:08:38,349 [I] DB_PATCHES have already been applied
I20260620 00:08:38.355762 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task8WXHPi/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 00:08:38.355866 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260620 00:08:38.359062 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:52519
Jun 20, 2026 12:08:38 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 12:08:38 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 12:08:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 12:08:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 12:08:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 12:08:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 12:08:39 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 00:08:39.477815 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.985s user 0.000s sys 0.010s
Jun 20, 2026 12:09:00 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 12:09:00 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 20, 2026 12:09:00 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 12:09:00 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 12:09:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 12:09:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 12:09:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 12:09:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 12:09:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 12:09:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 12:09:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 12:09:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 12:09:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 12:09:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 12:09:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 12:09:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 12:09:00 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 12:09:01 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 00:09:02.327788 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 00:09:02 dist-test-slave-grqt krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914142, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 00:09:02.410485 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 00:09:02.419842 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 00:09:03.300451 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 00:09:03.300545 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ranger-kms
I20260620 00:09:03.300567 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:41643
2026-06-20 00:09:03,463 [I] DB FLAVOR :POSTGRES
2026-06-20 00:09:03,463 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 00:09:03,464 [I] Checking connection
2026-06-20 00:09:03,464 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41643/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 00:09:03,717 [I] connection success
2026-06-20 00:09:03,717 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 00:09:03,717 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 00:09:03,717 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41643/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 00:09:03,952 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 00:09:03,952 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 00:09:03,953 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 00:09:03,953 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41643/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 00:09:04,213 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 00:09:04.666131 392 mini_ranger_kms.cc:310] Created kms service
I20260620 00:09:04.842167 392 mini_ranger_kms.cc:326] Created kudu user
I20260620 00:09:04.888290 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 00:09:04 dist-test-slave-grqt krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914142, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:09:05.304493 392 mini_ranger_kms.cc:384] Added ranger policy
I20260620 00:09:05.304705 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task8WXHPi/build/release/bin/postgresql.jar:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task8WXHPi/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 00:09:05.304741 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260620 00:09:05.308626 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12542
I20260620 00:09:05.308718 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:54749
Jun 20, 2026 12:09:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 12:09:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 12:09:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 12:09:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 12:09:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 12:09:07 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 00:09:07.592924 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.182s user 0.002s sys 0.143s
I20260620 00:09:07.593039 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 00:09:07.593084 392 mini_ranger_kms.cc:401] 127.0.98.20:54749/kms/v1/keys
Jun 20 00:09:07 dist-test-slave-grqt krb5kdc[12148](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781914147, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 00:09:08 dist-test-slave-grqt krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914147, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 00:09:08 dist-test-slave-grqt krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914147, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 00:09:10 dist-test-slave-grqt krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:09:10.897258 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42021
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42021
--ranger_config_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54749/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:09:10.980859 12592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:09:10.981115 12592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:09:10.981174 12592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:09:10.982533 12592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:09:10.982613 12592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:09:10.982638 12592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:09:10.982671 12592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:09:10.982698 12592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:09:10.984342 12592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:54749/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42021
--ranger_config_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:09:10.984642 12592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:09:10.984880 12592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:09:10.990391 12597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:09:10.990559 12592 server_base.cc:1034] running on GCE node
W20260620 00:09:10.990376 12598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:10.990343 12600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:09:10.990891 12592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:09:10.991247 12592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:09:10.992408 12592 hybrid_clock.cc:648] HybridClock initialized: now 1781914150992395 us; error 31 us; skew 500 ppm
Jun 20 00:09:10 dist-test-slave-grqt krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:09:10.995159 12592 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:09:10.995730 12592 webserver.cc:460] Webserver started at http://127.0.98.62:34905/ using document root <none> and password file <none>
I20260620 00:09:10.995966 12592 fs_manager.cc:359] Metadata directory not provided
I20260620 00:09:10.996026 12592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:09:10.996172 12592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 00:09:10 dist-test-slave-grqt krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:09:11.059295 12592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "ebfd1638ec78413c8a1c9b23d16bf956"
format_stamp: "Formatted at 2026-06-20 00:09:11 on dist-test-slave-grqt"
server_key: "2949d943034bbb84681a8bdf25c72101"
server_key_iv: "1145439cfe1712460f4c9722c464676d"
server_key_version: "kuduclusterkey@0"
I20260620 00:09:11.059759 12592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "ebfd1638ec78413c8a1c9b23d16bf956"
format_stamp: "Formatted at 2026-06-20 00:09:11 on dist-test-slave-grqt"
server_key: "2949d943034bbb84681a8bdf25c72101"
server_key_iv: "1145439cfe1712460f4c9722c464676d"
server_key_version: "kuduclusterkey@0"
I20260620 00:09:11.061201 12592 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260620 00:09:11.076035 12607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:11.076279 12592 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:09:11.076397 12592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "ebfd1638ec78413c8a1c9b23d16bf956"
format_stamp: "Formatted at 2026-06-20 00:09:11 on dist-test-slave-grqt"
server_key: "2949d943034bbb84681a8bdf25c72101"
server_key_iv: "1145439cfe1712460f4c9722c464676d"
server_key_version: "kuduclusterkey@0"
I20260620 00:09:11.076498 12592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:09:11.093578 12592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:09:11.096050 12592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:09:11.096226 12592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:09:11.101533 12592 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42021
I20260620 00:09:11.101560 12659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42021 every 8 connection(s)
I20260620 00:09:11.102056 12592 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:09:11.102828 12660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:09:11.103760 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 12592
I20260620 00:09:11.103885 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:09:11.104064 392 external_mini_cluster.cc:1428] Setting key 0363f369296191ae4230a1f50fed0b2b
I20260620 00:09:11.106149 12660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956: Bootstrap starting.
I20260620 00:09:11.107012 12660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956: Neither blocks nor log segments found. Creating new log.
I20260620 00:09:11.107393 12660 log.cc:824] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956: Log is configured to *not* fsync() on all Append() calls
I20260620 00:09:11.108260 12660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956: No bootstrap required, opened a new log
I20260620 00:09:11.110754 12660 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebfd1638ec78413c8a1c9b23d16bf956" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } }
I20260620 00:09:11.110987 12660 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:09:11.111047 12660 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebfd1638ec78413c8a1c9b23d16bf956, State: Initialized, Role: FOLLOWER
I20260620 00:09:11.111224 12660 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebfd1638ec78413c8a1c9b23d16bf956" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } }
I20260620 00:09:11.111344 12660 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:09:11.111400 12660 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:09:11.111470 12660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:09:11.112066 12660 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebfd1638ec78413c8a1c9b23d16bf956" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } }
I20260620 00:09:11.112265 12660 leader_election.cc:302] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebfd1638ec78413c8a1c9b23d16bf956; no voters:
I20260620 00:09:11.112501 12660 leader_election.cc:288] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:09:11.112533 12665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:09:11.112756 12665 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956 [term 1 LEADER]: Becoming Leader. State: Replica: ebfd1638ec78413c8a1c9b23d16bf956, State: Running, Role: LEADER
Jun 20 00:09:11 dist-test-slave-grqt krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:09:11.112903 12660 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:09:11.112920 12665 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebfd1638ec78413c8a1c9b23d16bf956" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } }
I20260620 00:09:11.113490 12666 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebfd1638ec78413c8a1c9b23d16bf956" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebfd1638ec78413c8a1c9b23d16bf956" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } }
I20260620 00:09:11.113646 12666 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956 [sys.catalog]: This master's current role is: LEADER
I20260620 00:09:11.113852 12666 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebfd1638ec78413c8a1c9b23d16bf956. Latest consensus state: current_term: 1 leader_uuid: "ebfd1638ec78413c8a1c9b23d16bf956" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebfd1638ec78413c8a1c9b23d16bf956" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } }
I20260620 00:09:11.113948 12666 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956 [sys.catalog]: This master's current role is: LEADER
I20260620 00:09:11.114099 12674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:09:11.114645 12674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:09:11.115103 12660 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 00:09:11.121320 12663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:11.104664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53428 (local address 127.0.98.62:42021)
0620 00:09:11.105521 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:11.105533 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:11.105571 (+ 38us) server_negotiation.cc:407] Connection header received
0620 00:09:11.106109 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:11.106119 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:11.106189 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:11.106390 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:09:11.107283 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:11.108612 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:11.109665 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:11.110313 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:11.110443 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:11.113408 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:09:11.113413 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:09:11.113414 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:09:11.113427 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:09:11.117760 (+ 4333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:11.118457 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:11.118461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:11.118461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:11.118712 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:11.118992 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:11.118992 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:11.118993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:11.119669 (+ 676us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:09:11.120407 (+ 738us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:11.120497 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 00:09:11.120952 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":789,"thread_start_us":100,"threads_started":1}
I20260620 00:09:11.121681 12681 mvcc.cc:205] Tried to move back new op lower bound from 7298720362981908480 to 7298720362959228928. Current Snapshot: MvccSnapshot[applied={T|T < 7298720362981908480}]
I20260620 00:09:11.125880 12674 catalog_manager.cc:1347] Generated new cluster ID: 02af59527afb40cd93713dbe7794eb00
I20260620 00:09:11.125948 12674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:09:11.138629 12674 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:09:11.139462 12674 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:09:11.147084 12674 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956: Generated new TSK 0
I20260620 00:09:11.147431 12674 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:09:11.889927 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54749/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42021
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:09:12.003235 12705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:09:12.003613 12705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:09:12.003665 12705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:09:12.005525 12705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:09:12.005592 12705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:09:12.005635 12705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:09:12.008001 12705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:54749/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:09:12.008291 12705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:09:12.008589 12705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:09:12.015604 12713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:12.015630 12710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:12.015612 12711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:09:12.015885 12705 server_base.cc:1034] running on GCE node
I20260620 00:09:12.016196 12705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:09:12.016494 12705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:09:12.017659 12705 hybrid_clock.cc:648] HybridClock initialized: now 1781914152017629 us; error 42 us; skew 500 ppm
Jun 20 00:09:12 dist-test-slave-grqt krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:09:12.020633 12705 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:09:12.021205 12705 webserver.cc:460] Webserver started at http://127.0.98.1:40979/ using document root <none> and password file <none>
I20260620 00:09:12.021526 12705 fs_manager.cc:359] Metadata directory not provided
I20260620 00:09:12.021600 12705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:09:12.021742 12705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 00:09:12 dist-test-slave-grqt krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:09:12.035130 12705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-0/data/instance:
uuid: "b29d70e53db24fdf9a0524ea2f97241f"
format_stamp: "Formatted at 2026-06-20 00:09:12 on dist-test-slave-grqt"
server_key: "662e552f9593b99df83086c3794cca9d"
server_key_iv: "bada0bf28890f068189088f79fee241c"
server_key_version: "kuduclusterkey@0"
I20260620 00:09:12.035604 12705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b29d70e53db24fdf9a0524ea2f97241f"
format_stamp: "Formatted at 2026-06-20 00:09:12 on dist-test-slave-grqt"
server_key: "662e552f9593b99df83086c3794cca9d"
server_key_iv: "bada0bf28890f068189088f79fee241c"
server_key_version: "kuduclusterkey@0"
I20260620 00:09:12.037618 12705 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 00:09:12.051173 12720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:12.051465 12705 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:09:12.051574 12705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-0/wal
uuid: "b29d70e53db24fdf9a0524ea2f97241f"
format_stamp: "Formatted at 2026-06-20 00:09:12 on dist-test-slave-grqt"
server_key: "662e552f9593b99df83086c3794cca9d"
server_key_iv: "bada0bf28890f068189088f79fee241c"
server_key_version: "kuduclusterkey@0"
I20260620 00:09:12.051697 12705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:09:12.062192 12705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:09:12.064723 12705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:09:12.064942 12705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:09:12.065223 12705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:09:12.065655 12705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:09:12.065747 12705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:12.065846 12705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:09:12.065920 12705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:12.072165 12705 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41743
I20260620 00:09:12.072679 12705 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:09:12.077720 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 12705
I20260620 00:09:12.077869 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:09:12.078011 12835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41743 every 8 connection(s)
I20260620 00:09:12.078089 392 external_mini_cluster.cc:1428] Setting key 4c047f05bfb993b7d21aace95366e0b7
Jun 20 00:09:12 dist-test-slave-grqt krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:09:12.088682 12839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:12.078593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56567 (local address 127.0.98.62:42021)
0620 00:09:12.078800 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:12.078802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:12.078876 (+ 74us) server_negotiation.cc:407] Connection header received
0620 00:09:12.079695 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:12.079697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:12.079711 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:12.079785 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:09:12.080617 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:12.081498 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:12.082784 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:12.083005 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:12.083095 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:12.085625 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:09:12.085627 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:09:12.085628 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:09:12.085639 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:09:12.087179 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:12.087641 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:12.087642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:12.087643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:12.087706 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:12.087974 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:12.087975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:12.087976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:12.088101 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:09:12.088179 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:12.088484 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 00:09:12.088574 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":67,"threads_started":1}
I20260620 00:09:12.088873 12838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:12.078515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.98.1:56567)
0620 00:09:12.078743 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:12.078803 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:12.079526 (+ 723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:12.079913 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:12.079919 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:12.080180 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:09:12.080484 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:12.080489 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:12.081614 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:12.081615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:12.082665 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:12.082668 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:12.083120 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:12.083121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:12.083350 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:12.084034 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:09:12.084043 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:09:12.085520 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:09:12.087288 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:12.087295 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:12.087297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:12.087545 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:12.087775 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:12.087776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:12.087777 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:12.087903 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:12.088171 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:09:12.088178 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:09:12.088340 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 00:09:12.088463 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 00:09:12.088618 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":98,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 00:09:12.089740 12836 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42021
I20260620 00:09:12.089867 12836 heartbeater.cc:464] Registering TS with master...
I20260620 00:09:12.090077 12836 heartbeater.cc:511] Master 127.0.98.62:42021 requested a full tablet report, sending...
I20260620 00:09:12.090682 12624 ts_manager.cc:194] Registered new tserver with Master: b29d70e53db24fdf9a0524ea2f97241f (127.0.98.1:41743)
I20260620 00:09:12.091380 12624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56567
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:09:12.108518 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54749/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42021
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:09:12.229046 12842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:09:12.229435 12842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:09:12.229488 12842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:09:12.231314 12842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:09:12.231380 12842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:09:12.231423 12842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:09:12.233748 12842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:54749/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:09:12.234032 12842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:09:12.234357 12842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:09:12.240708 12852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:12.240724 12850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:12.241186 12849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:09:12.241451 12842 server_base.cc:1034] running on GCE node
I20260620 00:09:12.241657 12842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:09:12.241931 12842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:09:12.243106 12842 hybrid_clock.cc:648] HybridClock initialized: now 1781914152243084 us; error 36 us; skew 500 ppm
Jun 20 00:09:12 dist-test-slave-grqt krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:09:12.246431 12842 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:09:12.247352 12842 webserver.cc:460] Webserver started at http://127.0.98.2:34627/ using document root <none> and password file <none>
I20260620 00:09:12.247659 12842 fs_manager.cc:359] Metadata directory not provided
I20260620 00:09:12.247742 12842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:09:12.247912 12842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 00:09:12 dist-test-slave-grqt krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:09:12.261869 12842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-1/data/instance:
uuid: "fd6890f7a16e4769a342b54b037eca8b"
format_stamp: "Formatted at 2026-06-20 00:09:12 on dist-test-slave-grqt"
server_key: "2afcfe61d7df451ea3bd4affe60cb725"
server_key_iv: "6860137bddae09b2d98aed2cf9f8c2eb"
server_key_version: "kuduclusterkey@0"
I20260620 00:09:12.262354 12842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fd6890f7a16e4769a342b54b037eca8b"
format_stamp: "Formatted at 2026-06-20 00:09:12 on dist-test-slave-grqt"
server_key: "2afcfe61d7df451ea3bd4affe60cb725"
server_key_iv: "6860137bddae09b2d98aed2cf9f8c2eb"
server_key_version: "kuduclusterkey@0"
I20260620 00:09:12.264935 12842 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
Jun 20 00:09:12 dist-test-slave-grqt krb5kdc[12148](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781914152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:09:12.282186 12859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:12.282450 12842 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.001s
I20260620 00:09:12.282550 12842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-1/wal
uuid: "fd6890f7a16e4769a342b54b037eca8b"
format_stamp: "Formatted at 2026-06-20 00:09:12 on dist-test-slave-grqt"
server_key: "2afcfe61d7df451ea3bd4affe60cb725"
server_key_iv: "6860137bddae09b2d98aed2cf9f8c2eb"
server_key_version: "kuduclusterkey@0"
I20260620 00:09:12.282653 12842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:09:12.305711 12842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:09:12.308454 12842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:09:12.308687 12842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:09:12.308975 12842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:09:12.309450 12842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:09:12.309547 12842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:12.309621 12842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:09:12.309676 12842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:12.316303 12842 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36535
I20260620 00:09:12.316850 12842 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:09:12.319392 12972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36535 every 8 connection(s)
I20260620 00:09:12.326301 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 12842
I20260620 00:09:12.326412 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:09:12.326614 392 external_mini_cluster.cc:1428] Setting key 00d6d44bfdf56f34899760d5cc269d0f
Jun 20 00:09:12 dist-test-slave-grqt krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:09:12.333611 12839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:12.319140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48291 (local address 127.0.98.62:42021)
0620 00:09:12.319190 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:12.319192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:12.320144 (+ 952us) server_negotiation.cc:407] Connection header received
0620 00:09:12.320883 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:12.320885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:12.320899 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:12.320930 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:09:12.321877 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:12.322655 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:12.324207 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:12.324529 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:12.324573 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:12.330007 (+ 5434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:09:12.330010 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:09:12.330010 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:09:12.330020 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:09:12.331841 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:12.332400 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:12.332401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:12.332401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:12.332463 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:12.332743 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:12.332744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:12.332744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:12.332904 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:09:12.332960 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:12.333407 (+ 447us) server_negotiation.cc:299] Negotiation successful
0620 00:09:12.333505 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 00:09:12.334430 12975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:12.318858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.98.2:48291)
0620 00:09:12.319987 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:12.320039 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:12.320737 (+ 698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:12.321125 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:12.321129 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:12.321447 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:09:12.321745 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:12.321751 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:12.322778 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:12.322779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:12.324093 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:12.324097 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:12.324839 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:12.324840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:12.324981 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:12.325658 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:09:12.325668 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:09:12.329871 (+ 4203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:09:12.331954 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:12.331965 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:12.331968 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:12.332303 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:12.332532 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:12.332532 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:12.332532 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:12.332670 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:12.333012 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:09:12.333019 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:09:12.333223 (+ 204us) client_negotiation.cc:769] Sending connection context
0620 00:09:12.334006 (+ 783us) client_negotiation.cc:240] Negotiation successful
0620 00:09:12.334186 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1081,"thread_start_us":82,"threads_started":1}
I20260620 00:09:12.335155 12973 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42021
I20260620 00:09:12.335250 12973 heartbeater.cc:464] Registering TS with master...
I20260620 00:09:12.335441 12973 heartbeater.cc:511] Master 127.0.98.62:42021 requested a full tablet report, sending...
I20260620 00:09:12.335889 12624 ts_manager.cc:194] Registered new tserver with Master: fd6890f7a16e4769a342b54b037eca8b (127.0.98.2:36535)
I20260620 00:09:12.336371 12624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48291
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:09:12.364897 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54749/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42021
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:09:12.484978 12979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:09:12.485292 12979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:09:12.485332 12979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:09:12.487229 12979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:09:12.487294 12979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:09:12.487339 12979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:09:12.489769 12979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:54749/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:09:12.490065 12979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:09:12.490404 12979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:09:12.497495 12987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:12.497490 12984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:12.497750 12985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:09:12.498138 12979 server_base.cc:1034] running on GCE node
I20260620 00:09:12.498347 12979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:09:12.498625 12979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:09:12.499799 12979 hybrid_clock.cc:648] HybridClock initialized: now 1781914152499773 us; error 36 us; skew 500 ppm
Jun 20 00:09:12 dist-test-slave-grqt krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:09:12.503432 12979 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:09:12.504020 12979 webserver.cc:460] Webserver started at http://127.0.98.3:44263/ using document root <none> and password file <none>
I20260620 00:09:12.504276 12979 fs_manager.cc:359] Metadata directory not provided
I20260620 00:09:12.504330 12979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:09:12.504478 12979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 00:09:12 dist-test-slave-grqt krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:09:12.525904 12979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-2/data/instance:
uuid: "75e2228c379b4351b4697d509a6fdbc4"
format_stamp: "Formatted at 2026-06-20 00:09:12 on dist-test-slave-grqt"
server_key: "d15bec67eae831aaed46728f6930eccd"
server_key_iv: "739d47f3e60e3a4246dc890318634cd4"
server_key_version: "kuduclusterkey@0"
I20260620 00:09:12.526378 12979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-2/wal/instance:
uuid: "75e2228c379b4351b4697d509a6fdbc4"
format_stamp: "Formatted at 2026-06-20 00:09:12 on dist-test-slave-grqt"
server_key: "d15bec67eae831aaed46728f6930eccd"
server_key_iv: "739d47f3e60e3a4246dc890318634cd4"
server_key_version: "kuduclusterkey@0"
I20260620 00:09:12.528146 12979 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 00:09:12.542728 12994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:12.542936 12979 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:09:12.543031 12979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-2/wal
uuid: "75e2228c379b4351b4697d509a6fdbc4"
format_stamp: "Formatted at 2026-06-20 00:09:12 on dist-test-slave-grqt"
server_key: "d15bec67eae831aaed46728f6930eccd"
server_key_iv: "739d47f3e60e3a4246dc890318634cd4"
server_key_version: "kuduclusterkey@0"
I20260620 00:09:12.543133 12979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:09:12.588919 12979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:09:12.591619 12979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:09:12.591866 12979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:09:12.592177 12979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:09:12.592609 12979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:09:12.592701 12979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:12.592783 12979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:09:12.592857 12979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:12.599596 12979 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36743
I20260620 00:09:12.599895 13107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36743 every 8 connection(s)
I20260620 00:09:12.600178 12979 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:09:12.604322 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 12979
I20260620 00:09:12.604467 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:09:12.604703 392 external_mini_cluster.cc:1428] Setting key fb71c64dc0c21b80c76c58a5431ac6e7
Jun 20 00:09:12 dist-test-slave-grqt krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:09:12.612002 12839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:12.601001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59179 (local address 127.0.98.62:42021)
0620 00:09:12.601050 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:12.601053 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:12.601341 (+ 288us) server_negotiation.cc:407] Connection header received
0620 00:09:12.602031 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:12.602032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:12.602045 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:12.602119 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:09:12.602870 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:12.603659 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:12.605352 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:12.605660 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:12.605715 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:12.608351 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:09:12.608353 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:09:12.608353 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:09:12.608361 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:09:12.610256 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:12.610793 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:12.610794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:12.610795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:12.610857 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:12.611174 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:12.611176 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:12.611176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:12.611335 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:09:12.611416 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:12.611794 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 00:09:12.611899 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 00:09:12.612332 13110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:12.600985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.98.3:59179)
0620 00:09:12.601173 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:12.601224 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:12.601894 (+ 670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:12.602200 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:12.602208 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:12.602462 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:09:12.602752 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:12.602759 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:12.603788 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:12.603789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:12.605265 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:12.605268 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:12.605818 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:12.605819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:12.605962 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:12.606602 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:09:12.606612 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:09:12.608236 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:09:12.610354 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:12.610364 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:12.610368 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:12.610700 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:12.610940 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:12.610941 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:12.610941 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:12.611092 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:12.611419 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:09:12.611425 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:09:12.611618 (+ 193us) client_negotiation.cc:769] Sending connection context
0620 00:09:12.611753 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 00:09:12.611925 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":92,"threads_started":1}
I20260620 00:09:12.613065 13108 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42021
I20260620 00:09:12.613165 13108 heartbeater.cc:464] Registering TS with master...
I20260620 00:09:12.613399 13108 heartbeater.cc:511] Master 127.0.98.62:42021 requested a full tablet report, sending...
I20260620 00:09:12.613876 12624 ts_manager.cc:194] Registered new tserver with Master: 75e2228c379b4351b4697d509a6fdbc4 (127.0.98.3:36743)
I20260620 00:09:12.614393 12624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59179
I20260620 00:09:12.616328 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:09:12.617007 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 12705
I20260620 00:09:12.624224 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 12842
I20260620 00:09:12.628443 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 12979
I20260620 00:09:12.633668 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 12592
I20260620 00:09:12.638723 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42021
--webserver_interface=127.0.98.62
--webserver_port=34905
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42021
--ranger_config_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54749/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:54749/kms with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:09:12.722764 13111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:09:12.723004 13111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:09:12.723029 13111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:09:12.724220 13111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:09:12.724267 13111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:09:12.724282 13111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:09:12.724294 13111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:09:12.724306 13111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:09:12.725975 13111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:54749/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42021
--ranger_config_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:09:12.726198 13111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:09:12.726388 13111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:09:12.731444 13116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:12.731468 13117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:12.731444 13119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:09:12.731763 13111 server_base.cc:1034] running on GCE node
I20260620 00:09:12.732009 13111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:09:12.732306 13111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:09:12.733471 13111 hybrid_clock.cc:648] HybridClock initialized: now 1781914152733462 us; error 38 us; skew 500 ppm
Jun 20 00:09:12 dist-test-slave-grqt krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:09:12.739869 13111 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:09:12.740418 13111 webserver.cc:460] Webserver started at http://127.0.98.62:34905/ using document root <none> and password file <none>
I20260620 00:09:12.740653 13111 fs_manager.cc:359] Metadata directory not provided
I20260620 00:09:12.740732 13111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 00:09:12 dist-test-slave-grqt krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:09:12.787487 13111 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 00:09:12.788405 13127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:12.788600 13111 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:09:12.788728 13111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "ebfd1638ec78413c8a1c9b23d16bf956"
format_stamp: "Formatted at 2026-06-20 00:09:11 on dist-test-slave-grqt"
server_key: "2949d943034bbb84681a8bdf25c72101"
server_key_iv: "1145439cfe1712460f4c9722c464676d"
server_key_version: "kuduclusterkey@0"
I20260620 00:09:12.789254 13111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:09:12.804836 13111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:09:12.806814 13111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:09:12.806943 13111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:09:12.811411 13111 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42021
I20260620 00:09:12.811444 13179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42021 every 8 connection(s)
I20260620 00:09:12.811849 13111 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:09:12.813098 13180 sys_catalog.cc:263] Verifying existing consensus state
I20260620 00:09:12.813805 13180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956: Bootstrap starting.
I20260620 00:09:12.814910 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 13111
I20260620 00:09:12.815255 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54749/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41743
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42021
--webserver_port=40979
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:54749/kms with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:09:12.822240 13180 log.cc:824] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956: Log is configured to *not* fsync() on all Append() calls
I20260620 00:09:12.824194 13180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 00:09:12.824532 13180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956: Bootstrap complete.
I20260620 00:09:12.827010 13180 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebfd1638ec78413c8a1c9b23d16bf956" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } }
I20260620 00:09:12.827248 13180 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebfd1638ec78413c8a1c9b23d16bf956, State: Initialized, Role: FOLLOWER
I20260620 00:09:12.827435 13180 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebfd1638ec78413c8a1c9b23d16bf956" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } }
I20260620 00:09:12.827503 13180 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:09:12.827553 13180 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:09:12.827602 13180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956 [term 1 FOLLOWER]: Advancing to term 2
I20260620 00:09:12.828696 13180 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebfd1638ec78413c8a1c9b23d16bf956" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } }
I20260620 00:09:12.828809 13180 leader_election.cc:302] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebfd1638ec78413c8a1c9b23d16bf956; no voters:
I20260620 00:09:12.829063 13180 leader_election.cc:288] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 00:09:12.829118 13184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 00:09:12.829255 13184 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956 [term 2 LEADER]: Becoming Leader. State: Replica: ebfd1638ec78413c8a1c9b23d16bf956, State: Running, Role: LEADER
I20260620 00:09:12.829421 13184 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebfd1638ec78413c8a1c9b23d16bf956" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } }
I20260620 00:09:12.829427 13180 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:09:12.830791 13185 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebfd1638ec78413c8a1c9b23d16bf956. Latest consensus state: current_term: 2 leader_uuid: "ebfd1638ec78413c8a1c9b23d16bf956" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebfd1638ec78413c8a1c9b23d16bf956" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } }
I20260620 00:09:12.830991 13185 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956 [sys.catalog]: This master's current role is: LEADER
I20260620 00:09:12.831125 13186 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ebfd1638ec78413c8a1c9b23d16bf956" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebfd1638ec78413c8a1c9b23d16bf956" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } }
I20260620 00:09:12.831252 13199 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:09:12.831275 13186 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956 [sys.catalog]: This master's current role is: LEADER
I20260620 00:09:12.831184 13180 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 00:09:12.834241 13199 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:09:12.834513 13199 catalog_manager.cc:1259] Loaded cluster ID: 02af59527afb40cd93713dbe7794eb00
I20260620 00:09:12.834561 13199 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:09:12.836153 13199 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:09:12.836378 13199 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956: Loaded TSK: 0
I20260620 00:09:12.836699 13199 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 00:09:12.928861 13182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:09:12.929257 13182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:09:12.929384 13182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:09:12.931290 13182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:09:12.931358 13182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:09:12.931401 13182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:09:12.933779 13182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:54749/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41743
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40979
--tserver_master_addrs=127.0.98.62:42021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:09:12.934031 13182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:09:12.934325 13182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:09:12.942648 13217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:12.942648 13220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:12.942809 13218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:09:12.945575 13182 server_base.cc:1034] running on GCE node
I20260620 00:09:12.945937 13182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:09:12.946238 13182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:09:12.947423 13182 hybrid_clock.cc:648] HybridClock initialized: now 1781914152947401 us; error 37 us; skew 500 ppm
Jun 20 00:09:12 dist-test-slave-grqt krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:09:12.950738 13182 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:09:12.951314 13182 webserver.cc:460] Webserver started at http://127.0.98.1:40979/ using document root <none> and password file <none>
I20260620 00:09:12.951593 13182 fs_manager.cc:359] Metadata directory not provided
I20260620 00:09:12.951678 13182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 00:09:12 dist-test-slave-grqt krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:09:12.981704 13182 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.001s sys 0.000s
I20260620 00:09:12.982586 13228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:12.982782 13182 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:09:12.982862 13182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-0/wal
uuid: "b29d70e53db24fdf9a0524ea2f97241f"
format_stamp: "Formatted at 2026-06-20 00:09:12 on dist-test-slave-grqt"
server_key: "662e552f9593b99df83086c3794cca9d"
server_key_iv: "bada0bf28890f068189088f79fee241c"
server_key_version: "kuduclusterkey@0"
I20260620 00:09:12.983318 13182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:09:13.011214 13182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:09:13.013995 13182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:09:13.014235 13182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:09:13.014595 13182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:09:13.015054 13182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:09:13.015151 13182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:13.015270 13182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:09:13.015358 13182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:13.021863 13182 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41743
I20260620 00:09:13.022435 13182 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:09:13.031952 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 13182
I20260620 00:09:13.032279 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54749/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36535
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42021
--webserver_port=34627
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:54749/kms with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 00:09:13 dist-test-slave-grqt krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:09:13.037042 13341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41743 every 8 connection(s)
I20260620 00:09:13.043251 13344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:13.023630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.98.1:48407)
0620 00:09:13.024223 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:13.024272 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:13.025472 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:13.026259 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:13.026262 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:13.026526 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:09:13.026814 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:13.026821 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:13.028474 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:13.028475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:13.029876 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:13.029879 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:13.030879 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:13.030880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:13.031027 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:13.031645 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:09:13.031655 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:09:13.034840 (+ 3185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:09:13.040942 (+ 6102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:13.040950 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:13.040953 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:13.041306 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:13.041641 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:13.041642 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:13.041642 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:13.041800 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:13.042342 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:09:13.042351 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:09:13.042723 (+ 372us) client_negotiation.cc:769] Sending connection context
0620 00:09:13.042832 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 00:09:13.043004 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":88,"threads_started":1}
I20260620 00:09:13.044235 13345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:13.024005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48407 (local address 127.0.98.62:42021)
0620 00:09:13.025698 (+ 1693us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:13.025707 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:13.025747 (+ 40us) server_negotiation.cc:407] Connection header received
0620 00:09:13.025897 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:13.025903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:13.025959 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:13.026091 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:09:13.026956 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:13.028344 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:13.029978 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:13.030652 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:13.030766 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:13.034994 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:09:13.034999 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:09:13.035000 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:09:13.038524 (+ 3524us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:09:13.040821 (+ 2297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:13.041432 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:13.041436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:13.041437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:13.041556 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:13.041883 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:13.041884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:13.041885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:13.042227 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:09:13.042311 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:13.043907 (+ 1596us) server_negotiation.cc:299] Negotiation successful
0620 00:09:13.044041 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1630,"thread_start_us":75,"threads_started":1}
I20260620 00:09:13.045156 13342 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42021
I20260620 00:09:13.045254 13342 heartbeater.cc:464] Registering TS with master...
I20260620 00:09:13.045475 13342 heartbeater.cc:511] Master 127.0.98.62:42021 requested a full tablet report, sending...
W20260620 00:09:13.046388 13342 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42021 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 00:09:13.046463 13342 heartbeater.cc:464] Registering TS with master...
I20260620 00:09:13.046530 13342 heartbeater.cc:511] Master 127.0.98.62:42021 requested a full tablet report, sending...
I20260620 00:09:13.046947 13342 heartbeater.cc:464] Registering TS with master...
I20260620 00:09:13.047024 13342 heartbeater.cc:511] Master 127.0.98.62:42021 requested a full tablet report, sending...
W20260620 00:09:13.047348 13342 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 00:09:13.144951 13346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:09:13.145210 13346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:09:13.145253 13346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:09:13.146596 13346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:09:13.146677 13346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:09:13.146727 13346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:09:13.148387 13346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:54749/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36535
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34627
--tserver_master_addrs=127.0.98.62:42021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:09:13.148656 13346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:09:13.148923 13346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:09:13.155418 13351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:13.155427 13352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:09:13.155845 13346 server_base.cc:1034] running on GCE node
W20260620 00:09:13.155514 13354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:09:13.156210 13346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:09:13.156497 13346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:09:13.157688 13346 hybrid_clock.cc:648] HybridClock initialized: now 1781914153157671 us; error 37 us; skew 500 ppm
Jun 20 00:09:13 dist-test-slave-grqt krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:09:13.160884 13346 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:09:13.161468 13346 webserver.cc:460] Webserver started at http://127.0.98.2:34627/ using document root <none> and password file <none>
I20260620 00:09:13.161726 13346 fs_manager.cc:359] Metadata directory not provided
I20260620 00:09:13.161811 13346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 00:09:13 dist-test-slave-grqt krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:09:13.179548 13346 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:09:13.180439 13362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:13.180594 13346 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:09:13.180676 13346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-1/wal
uuid: "fd6890f7a16e4769a342b54b037eca8b"
format_stamp: "Formatted at 2026-06-20 00:09:12 on dist-test-slave-grqt"
server_key: "2afcfe61d7df451ea3bd4affe60cb725"
server_key_iv: "6860137bddae09b2d98aed2cf9f8c2eb"
server_key_version: "kuduclusterkey@0"
I20260620 00:09:13.181135 13346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:09:13.195685 13346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:09:13.197896 13346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:09:13.198041 13346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:09:13.198279 13346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:09:13.198614 13346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:09:13.198669 13346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:13.198711 13346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:09:13.198752 13346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:13.205109 13346 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36535
I20260620 00:09:13.205178 13475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36535 every 8 connection(s)
I20260620 00:09:13.205647 13346 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:09:13.206233 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 13346
I20260620 00:09:13.206542 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54749/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36743
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42021
--webserver_port=44263
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:54749/kms with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 00:09:13 dist-test-slave-grqt krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:09:13.218027 13345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:13.207732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47031 (local address 127.0.98.62:42021)
0620 00:09:13.207786 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:13.207788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:13.208088 (+ 300us) server_negotiation.cc:407] Connection header received
0620 00:09:13.208631 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:13.208632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:13.208650 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:13.208700 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:09:13.210248 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:13.210813 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:13.212400 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:13.212704 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:13.212751 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:13.215088 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:09:13.215089 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:09:13.215090 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:09:13.215098 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:09:13.216525 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:13.216968 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:13.216970 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:13.216970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:13.217028 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:13.217281 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:13.217282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:13.217282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:13.217458 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:09:13.217539 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:13.217833 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 00:09:13.217931 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 00:09:13.218222 13479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:13.207756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.98.2:47031)
0620 00:09:13.207958 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:13.208004 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:13.208504 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:13.209556 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:13.209561 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:13.209828 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:09:13.210135 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:13.210142 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:13.210937 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:13.210938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:13.212269 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:13.212272 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:13.212863 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:13.212864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:13.212983 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:13.213634 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:09:13.213643 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:09:13.214986 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:09:13.216626 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:13.216634 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:13.216636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:13.216880 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:13.217094 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:13.217095 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:13.217096 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:13.217213 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:13.217531 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:09:13.217537 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:09:13.217700 (+ 163us) client_negotiation.cc:769] Sending connection context
0620 00:09:13.217815 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 00:09:13.217968 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":129,"threads_started":1}
I20260620 00:09:13.218773 13476 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42021
I20260620 00:09:13.218904 13476 heartbeater.cc:464] Registering TS with master...
I20260620 00:09:13.219092 13476 heartbeater.cc:511] Master 127.0.98.62:42021 requested a full tablet report, sending...
W20260620 00:09:13.219748 13476 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42021 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 00:09:13.219831 13476 heartbeater.cc:464] Registering TS with master...
I20260620 00:09:13.219904 13476 heartbeater.cc:511] Master 127.0.98.62:42021 requested a full tablet report, sending...
I20260620 00:09:13.220227 13476 heartbeater.cc:464] Registering TS with master...
I20260620 00:09:13.220300 13476 heartbeater.cc:511] Master 127.0.98.62:42021 requested a full tablet report, sending...
W20260620 00:09:13.220569 13476 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 00:09:13.331969 13478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:09:13.332266 13478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:09:13.332305 13478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:09:13.334208 13478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:09:13.334282 13478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:09:13.334328 13478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:09:13.336771 13478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:54749/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36743
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44263
--tserver_master_addrs=127.0.98.62:42021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:09:13.337059 13478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:09:13.337387 13478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:09:13.343281 13486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:13.343281 13485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:13.343381 13488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:09:13.343299 13478 server_base.cc:1034] running on GCE node
I20260620 00:09:13.343694 13478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:09:13.344040 13478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:09:13.347399 13478 hybrid_clock.cc:648] HybridClock initialized: now 1781914153347362 us; error 59 us; skew 500 ppm
Jun 20 00:09:13 dist-test-slave-grqt krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:09:13.350862 13478 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:09:13.351475 13478 webserver.cc:460] Webserver started at http://127.0.98.3:44263/ using document root <none> and password file <none>
I20260620 00:09:13.351747 13478 fs_manager.cc:359] Metadata directory not provided
I20260620 00:09:13.351807 13478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 00:09:13 dist-test-slave-grqt krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:09:13.369803 13478 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:09:13.370654 13496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:13.370951 13478 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:09:13.371078 13478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-2/wal
uuid: "75e2228c379b4351b4697d509a6fdbc4"
format_stamp: "Formatted at 2026-06-20 00:09:12 on dist-test-slave-grqt"
server_key: "d15bec67eae831aaed46728f6930eccd"
server_key_iv: "739d47f3e60e3a4246dc890318634cd4"
server_key_version: "kuduclusterkey@0"
I20260620 00:09:13.371477 13478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:09:13.385416 13478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:09:13.387921 13478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:09:13.388099 13478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:09:13.388370 13478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:09:13.388783 13478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:09:13.388839 13478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:13.388907 13478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:09:13.388937 13478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:13.396337 13478 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36743
I20260620 00:09:13.396438 13609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36743 every 8 connection(s)
I20260620 00:09:13.396908 13478 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 00:09:13 dist-test-slave-grqt krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:09:13.404624 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 13478
I20260620 00:09:13.407846 13612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:13.397794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.98.3:43023)
0620 00:09:13.397975 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:13.398020 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:13.398536 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:13.398810 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:13.398814 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:13.399024 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:09:13.399241 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:13.399246 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:13.399981 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:13.399982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:13.400835 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:13.400838 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:13.401302 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:13.401303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:13.401510 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:13.402115 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:09:13.402124 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:09:13.403541 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:09:13.406074 (+ 2533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:13.406085 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:13.406088 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:13.406447 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:13.406681 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:13.406681 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:13.406682 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:13.406829 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:13.407099 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:09:13.407104 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:09:13.407294 (+ 190us) client_negotiation.cc:769] Sending connection context
0620 00:09:13.407410 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 00:09:13.407577 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":110,"threads_started":1}
I20260620 00:09:13.408556 13345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:13.397760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43023 (local address 127.0.98.62:42021)
0620 00:09:13.397811 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:13.397813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:13.398101 (+ 288us) server_negotiation.cc:407] Connection header received
0620 00:09:13.398662 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:13.398664 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:13.398677 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:13.398726 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:09:13.399358 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:13.399867 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:13.400941 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:13.401171 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:13.401263 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:13.403693 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:09:13.403695 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:09:13.403695 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:09:13.403703 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:09:13.405973 (+ 2270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:13.406523 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:13.406524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:13.406524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:13.406604 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:13.406900 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:13.406901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:13.406901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:13.407020 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:09:13.407075 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:13.408384 (+ 1309us) server_negotiation.cc:299] Negotiation successful
0620 00:09:13.408466 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 00:09:13.409127 13610 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42021
I20260620 00:09:13.409268 13610 heartbeater.cc:464] Registering TS with master...
I20260620 00:09:13.409499 13610 heartbeater.cc:511] Master 127.0.98.62:42021 requested a full tablet report, sending...
W20260620 00:09:13.410218 13610 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42021 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 00:09:13.410295 13610 heartbeater.cc:464] Registering TS with master...
I20260620 00:09:13.410359 13610 heartbeater.cc:511] Master 127.0.98.62:42021 requested a full tablet report, sending...
I20260620 00:09:13.410763 13610 heartbeater.cc:464] Registering TS with master...
I20260620 00:09:13.410842 13610 heartbeater.cc:511] Master 127.0.98.62:42021 requested a full tablet report, sending...
W20260620 00:09:13.411176 13610 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 00:09:13.413874 13613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:13.405084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53434 (local address 127.0.98.62:42021)
0620 00:09:13.405474 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:13.405476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:13.405483 (+ 7us) server_negotiation.cc:407] Connection header received
0620 00:09:13.405509 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:13.405511 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:13.405523 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:13.405580 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:09:13.406421 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:13.407263 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:13.408366 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:13.408670 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:13.408732 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:13.411107 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:09:13.411108 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:09:13.411108 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:09:13.411116 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:09:13.412643 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:13.413018 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:13.413019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:13.413019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:13.413079 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:13.413279 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:13.413280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:13.413281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:13.413453 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:09:13.413641 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:13.413677 (+ 36us) server_negotiation.cc:299] Negotiation successful
0620 00:09:13.413760 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":92,"threads_started":1}
Jun 20 00:09:13 dist-test-slave-grqt krb5kdc[12148](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781914153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:09:14.047546 13342 heartbeater.cc:464] Registering TS with master...
I20260620 00:09:14.047711 13342 heartbeater.cc:511] Master 127.0.98.62:42021 requested a full tablet report, sending...
I20260620 00:09:14.048252 13144 ts_manager.cc:194] Registered new tserver with Master: b29d70e53db24fdf9a0524ea2f97241f (127.0.98.1:41743)
I20260620 00:09:14.049328 13144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48407
I20260620 00:09:14.220805 13476 heartbeater.cc:464] Registering TS with master...
I20260620 00:09:14.221001 13476 heartbeater.cc:511] Master 127.0.98.62:42021 requested a full tablet report, sending...
I20260620 00:09:14.221454 13144 ts_manager.cc:194] Registered new tserver with Master: fd6890f7a16e4769a342b54b037eca8b (127.0.98.2:36535)
I20260620 00:09:14.222013 13144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47031
Jun 20 00:09:14 dist-test-slave-grqt krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:09:14.411335 13610 heartbeater.cc:464] Registering TS with master...
I20260620 00:09:14.411489 13610 heartbeater.cc:511] Master 127.0.98.62:42021 requested a full tablet report, sending...
I20260620 00:09:14.411819 13144 ts_manager.cc:194] Registered new tserver with Master: 75e2228c379b4351b4697d509a6fdbc4 (127.0.98.3:36743)
I20260620 00:09:14.412372 13144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43023
I20260620 00:09:14.420971 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:09:14.430336 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:14.422028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53442 (local address 127.0.98.62:42021)
0620 00:09:14.422298 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:14.422300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:14.422458 (+ 158us) server_negotiation.cc:407] Connection header received
0620 00:09:14.422482 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:14.422483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:14.422494 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:14.422548 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:09:14.423260 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:14.424045 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:14.425115 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:14.425473 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:14.425532 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:14.427036 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:09:14.427038 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:09:14.427038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:09:14.427046 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:09:14.428815 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:14.429242 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:14.429244 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:14.429244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:14.429300 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:14.429511 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:14.429512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:14.429512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:14.429722 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:09:14.429846 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:14.430143 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 00:09:14.430230 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":67,"threads_started":1}
I20260620 00:09:14.433137 13144 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53442:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:09:14.435567 13144 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:09:14.451896 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:14.444039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36743 (local address 127.0.0.1:39322)
0620 00:09:14.445865 (+ 1826us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:14.445875 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:14.445930 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:14.446190 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:14.446193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:14.446195 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:09:14.446553 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:14.446556 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:14.447832 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:14.447833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:14.449477 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:14.449480 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:14.451577 (+ 2097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:14.451578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:14.451666 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:14.451683 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 00:09:14.451750 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 00:09:14.451803 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1809,"thread_start_us":77,"threads_started":1}
I20260620 00:09:14.452356 13639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:14.444137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39322 (local address 127.0.98.3:36743)
0620 00:09:14.445807 (+ 1670us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:14.445812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:14.445985 (+ 173us) server_negotiation.cc:407] Connection header received
0620 00:09:14.446022 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:14.446024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:14.446039 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:14.446104 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:09:14.446632 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:14.447738 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:14.449569 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:14.450310 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:14.450380 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:14.450457 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:14.452225 (+ 1768us) server_negotiation.cc:299] Negotiation successful
0620 00:09:14.452277 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1632,"thread_start_us":58,"threads_started":1}
I20260620 00:09:14.454107 13528 tablet_service.cc:1467] Processing CreateTablet for tablet 4ef248707f044554812345a8122761ea (DEFAULT_TABLE table=test-table [id=41018e443bb949d3b762c7399fef77df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:09:14.454506 13528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ef248707f044554812345a8122761ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:09:14.456287 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:14.444024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36535 (local address 127.0.0.1:44664)
0620 00:09:14.444711 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:14.444753 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:14.444813 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:14.450845 (+ 6032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:14.450846 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:14.450847 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:09:14.450889 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:14.450890 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:14.453629 (+ 2739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:14.453630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:14.455077 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:14.455080 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:14.456053 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:14.456054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:14.456122 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:14.456127 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:09:14.456156 (+ 29us) client_negotiation.cc:240] Negotiation successful
0620 00:09:14.456197 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"mutex_wait_us":1,"thread_start_us":76,"threads_started":1}
I20260620 00:09:14.456525 13637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:14.444272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41743 (local address 127.0.0.1:54814)
0620 00:09:14.450514 (+ 6242us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:14.450525 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:14.450566 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:14.452941 (+ 2375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:14.452942 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:14.452945 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:09:14.452984 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:14.452986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:14.454203 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:14.454204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:14.455277 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:14.455280 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:14.456254 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:14.456255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:14.456335 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:14.456340 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:09:14.456391 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 00:09:14.456430 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6224,"thread_start_us":55,"threads_started":1}
I20260620 00:09:14.456689 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:14.444284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44664 (local address 127.0.98.2:36535)
0620 00:09:14.450682 (+ 6398us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:14.450687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:14.450698 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:09:14.450735 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:14.450737 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:14.450753 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:14.450950 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:09:14.451021 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:14.453533 (+ 2512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:14.455168 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:14.455952 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:14.456453 (+ 501us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:14.456548 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:14.456575 (+ 27us) server_negotiation.cc:299] Negotiation successful
0620 00:09:14.456616 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6361,"thread_start_us":66,"threads_started":1}
I20260620 00:09:14.456766 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:14.444362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54814 (local address 127.0.98.1:41743)
0620 00:09:14.452696 (+ 8334us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:14.452701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:14.452713 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:09:14.452748 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:14.452750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:14.452766 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:14.452852 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:09:14.453057 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:14.454103 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:14.455380 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:14.456083 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:14.456152 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:14.456216 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:14.456628 (+ 412us) server_negotiation.cc:299] Negotiation successful
0620 00:09:14.456670 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8285,"thread_start_us":53,"threads_started":1}
I20260620 00:09:14.457381 13410 tablet_service.cc:1467] Processing CreateTablet for tablet 4ef248707f044554812345a8122761ea (DEFAULT_TABLE table=test-table [id=41018e443bb949d3b762c7399fef77df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:09:14.457597 13268 tablet_service.cc:1467] Processing CreateTablet for tablet 4ef248707f044554812345a8122761ea (DEFAULT_TABLE table=test-table [id=41018e443bb949d3b762c7399fef77df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:09:14.457767 13410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ef248707f044554812345a8122761ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:09:14.457953 13268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ef248707f044554812345a8122761ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:09:14.458479 13643 tablet_bootstrap.cc:492] T 4ef248707f044554812345a8122761ea P 75e2228c379b4351b4697d509a6fdbc4: Bootstrap starting.
I20260620 00:09:14.459120 13643 tablet_bootstrap.cc:654] T 4ef248707f044554812345a8122761ea P 75e2228c379b4351b4697d509a6fdbc4: Neither blocks nor log segments found. Creating new log.
I20260620 00:09:14.459427 13643 log.cc:824] T 4ef248707f044554812345a8122761ea P 75e2228c379b4351b4697d509a6fdbc4: Log is configured to *not* fsync() on all Append() calls
I20260620 00:09:14.460150 13643 tablet_bootstrap.cc:492] T 4ef248707f044554812345a8122761ea P 75e2228c379b4351b4697d509a6fdbc4: No bootstrap required, opened a new log
I20260620 00:09:14.460217 13643 ts_tablet_manager.cc:1397] T 4ef248707f044554812345a8122761ea P 75e2228c379b4351b4697d509a6fdbc4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:09:14.462771 13643 raft_consensus.cc:348] T 4ef248707f044554812345a8122761ea P 75e2228c379b4351b4697d509a6fdbc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd6890f7a16e4769a342b54b037eca8b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36535 } } peers { permanent_uuid: "75e2228c379b4351b4697d509a6fdbc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36743 } } peers { permanent_uuid: "b29d70e53db24fdf9a0524ea2f97241f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41743 } }
I20260620 00:09:14.462939 13643 raft_consensus.cc:374] T 4ef248707f044554812345a8122761ea P 75e2228c379b4351b4697d509a6fdbc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:09:14.462970 13643 raft_consensus.cc:729] T 4ef248707f044554812345a8122761ea P 75e2228c379b4351b4697d509a6fdbc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75e2228c379b4351b4697d509a6fdbc4, State: Initialized, Role: FOLLOWER
I20260620 00:09:14.463253 13643 consensus_queue.cc:260] T 4ef248707f044554812345a8122761ea P 75e2228c379b4351b4697d509a6fdbc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd6890f7a16e4769a342b54b037eca8b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36535 } } peers { permanent_uuid: "75e2228c379b4351b4697d509a6fdbc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36743 } } peers { permanent_uuid: "b29d70e53db24fdf9a0524ea2f97241f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41743 } }
I20260620 00:09:14.463500 13643 ts_tablet_manager.cc:1428] T 4ef248707f044554812345a8122761ea P 75e2228c379b4351b4697d509a6fdbc4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:09:14.463697 13610 heartbeater.cc:503] Master 127.0.98.62:42021 was elected leader, sending a full tablet report...
I20260620 00:09:14.464996 13645 tablet_bootstrap.cc:492] T 4ef248707f044554812345a8122761ea P fd6890f7a16e4769a342b54b037eca8b: Bootstrap starting.
I20260620 00:09:14.466149 13645 tablet_bootstrap.cc:654] T 4ef248707f044554812345a8122761ea P fd6890f7a16e4769a342b54b037eca8b: Neither blocks nor log segments found. Creating new log.
I20260620 00:09:14.466506 13645 log.cc:824] T 4ef248707f044554812345a8122761ea P fd6890f7a16e4769a342b54b037eca8b: Log is configured to *not* fsync() on all Append() calls
I20260620 00:09:14.467286 13645 tablet_bootstrap.cc:492] T 4ef248707f044554812345a8122761ea P fd6890f7a16e4769a342b54b037eca8b: No bootstrap required, opened a new log
I20260620 00:09:14.467357 13645 ts_tablet_manager.cc:1397] T 4ef248707f044554812345a8122761ea P fd6890f7a16e4769a342b54b037eca8b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 00:09:14.469856 13645 raft_consensus.cc:348] T 4ef248707f044554812345a8122761ea P fd6890f7a16e4769a342b54b037eca8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd6890f7a16e4769a342b54b037eca8b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36535 } } peers { permanent_uuid: "75e2228c379b4351b4697d509a6fdbc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36743 } } peers { permanent_uuid: "b29d70e53db24fdf9a0524ea2f97241f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41743 } }
I20260620 00:09:14.470016 13645 raft_consensus.cc:374] T 4ef248707f044554812345a8122761ea P fd6890f7a16e4769a342b54b037eca8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:09:14.470044 13645 raft_consensus.cc:729] T 4ef248707f044554812345a8122761ea P fd6890f7a16e4769a342b54b037eca8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd6890f7a16e4769a342b54b037eca8b, State: Initialized, Role: FOLLOWER
I20260620 00:09:14.470376 13645 consensus_queue.cc:260] T 4ef248707f044554812345a8122761ea P fd6890f7a16e4769a342b54b037eca8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd6890f7a16e4769a342b54b037eca8b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36535 } } peers { permanent_uuid: "75e2228c379b4351b4697d509a6fdbc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36743 } } peers { permanent_uuid: "b29d70e53db24fdf9a0524ea2f97241f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41743 } }
I20260620 00:09:14.470949 13645 ts_tablet_manager.cc:1428] T 4ef248707f044554812345a8122761ea P fd6890f7a16e4769a342b54b037eca8b: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260620 00:09:14.471112 13476 heartbeater.cc:503] Master 127.0.98.62:42021 was elected leader, sending a full tablet report...
I20260620 00:09:14.471658 13646 tablet_bootstrap.cc:492] T 4ef248707f044554812345a8122761ea P b29d70e53db24fdf9a0524ea2f97241f: Bootstrap starting.
I20260620 00:09:14.472414 13646 tablet_bootstrap.cc:654] T 4ef248707f044554812345a8122761ea P b29d70e53db24fdf9a0524ea2f97241f: Neither blocks nor log segments found. Creating new log.
I20260620 00:09:14.472791 13646 log.cc:824] T 4ef248707f044554812345a8122761ea P b29d70e53db24fdf9a0524ea2f97241f: Log is configured to *not* fsync() on all Append() calls
I20260620 00:09:14.473768 13646 tablet_bootstrap.cc:492] T 4ef248707f044554812345a8122761ea P b29d70e53db24fdf9a0524ea2f97241f: No bootstrap required, opened a new log
I20260620 00:09:14.473832 13646 ts_tablet_manager.cc:1397] T 4ef248707f044554812345a8122761ea P b29d70e53db24fdf9a0524ea2f97241f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:09:14.476258 13646 raft_consensus.cc:348] T 4ef248707f044554812345a8122761ea P b29d70e53db24fdf9a0524ea2f97241f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd6890f7a16e4769a342b54b037eca8b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36535 } } peers { permanent_uuid: "75e2228c379b4351b4697d509a6fdbc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36743 } } peers { permanent_uuid: "b29d70e53db24fdf9a0524ea2f97241f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41743 } }
I20260620 00:09:14.476423 13646 raft_consensus.cc:374] T 4ef248707f044554812345a8122761ea P b29d70e53db24fdf9a0524ea2f97241f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:09:14.476454 13646 raft_consensus.cc:729] T 4ef248707f044554812345a8122761ea P b29d70e53db24fdf9a0524ea2f97241f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b29d70e53db24fdf9a0524ea2f97241f, State: Initialized, Role: FOLLOWER
I20260620 00:09:14.476760 13646 consensus_queue.cc:260] T 4ef248707f044554812345a8122761ea P b29d70e53db24fdf9a0524ea2f97241f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd6890f7a16e4769a342b54b037eca8b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36535 } } peers { permanent_uuid: "75e2228c379b4351b4697d509a6fdbc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36743 } } peers { permanent_uuid: "b29d70e53db24fdf9a0524ea2f97241f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41743 } }
I20260620 00:09:14.477111 13646 ts_tablet_manager.cc:1428] T 4ef248707f044554812345a8122761ea P b29d70e53db24fdf9a0524ea2f97241f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:09:14.477283 13342 heartbeater.cc:503] Master 127.0.98.62:42021 was elected leader, sending a full tablet report...
W20260620 00:09:14.524099 13343 tablet.cc:2369] T 4ef248707f044554812345a8122761ea P b29d70e53db24fdf9a0524ea2f97241f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:09:14.531704 13649 raft_consensus.cc:482] T 4ef248707f044554812345a8122761ea P fd6890f7a16e4769a342b54b037eca8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:09:14.531864 13649 raft_consensus.cc:504] T 4ef248707f044554812345a8122761ea P fd6890f7a16e4769a342b54b037eca8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd6890f7a16e4769a342b54b037eca8b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36535 } } peers { permanent_uuid: "75e2228c379b4351b4697d509a6fdbc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36743 } } peers { permanent_uuid: "b29d70e53db24fdf9a0524ea2f97241f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41743 } }
I20260620 00:09:14.532270 13649 leader_election.cc:288] T 4ef248707f044554812345a8122761ea P fd6890f7a16e4769a342b54b037eca8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75e2228c379b4351b4697d509a6fdbc4 (127.0.98.3:36743), b29d70e53db24fdf9a0524ea2f97241f (127.0.98.1:41743)
I20260620 00:09:14.536578 13639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:14.532670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54563 (local address 127.0.98.3:36743)
0620 00:09:14.532717 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:14.532719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:14.532860 (+ 141us) server_negotiation.cc:407] Connection header received
0620 00:09:14.532891 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:14.532892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:14.532904 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:14.532964 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:09:14.533213 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:14.533943 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:14.535654 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:14.536208 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:14.536253 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:14.536264 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:14.536468 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 00:09:14.536507 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 00:09:14.536695 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:14.532802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53603 (local address 127.0.98.1:41743)
0620 00:09:14.532843 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:14.532845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:14.532948 (+ 103us) server_negotiation.cc:407] Connection header received
0620 00:09:14.533044 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:14.533045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:14.533056 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:14.533087 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:09:14.533293 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:14.534072 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:14.535705 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:14.536272 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:14.536321 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:14.536330 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:14.536570 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 00:09:14.536611 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 00:09:14.537004 13652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:14.532486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41743 (local address 127.0.98.2:53603)
0620 00:09:14.532899 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:14.532906 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:14.532970 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:14.533149 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:14.533150 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:14.533151 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:09:14.533219 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:14.533221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:14.534172 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:14.534173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:14.535583 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:14.535586 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:14.536406 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:14.536407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:14.536488 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:14.536491 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:09:14.536869 (+ 378us) client_negotiation.cc:240] Negotiation successful
0620 00:09:14.536913 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":78,"threads_started":1}
I20260620 00:09:14.537112 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:14.532457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36743 (local address 127.0.98.2:54563)
0620 00:09:14.532736 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:14.532748 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:14.532817 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:14.533021 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:14.533022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:14.533024 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:09:14.533118 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:14.533121 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:14.534043 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:14.534044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:14.535553 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:14.535556 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:14.536320 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:14.536321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:14.536395 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:14.536398 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:09:14.536932 (+ 534us) client_negotiation.cc:240] Negotiation successful
0620 00:09:14.536977 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":98,"threads_started":1}
I20260620 00:09:14.537511 13277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef248707f044554812345a8122761ea" candidate_uuid: "fd6890f7a16e4769a342b54b037eca8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b29d70e53db24fdf9a0524ea2f97241f" is_pre_election: true
I20260620 00:09:14.537696 13277 raft_consensus.cc:2454] T 4ef248707f044554812345a8122761ea P b29d70e53db24fdf9a0524ea2f97241f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd6890f7a16e4769a342b54b037eca8b in term 0.
I20260620 00:09:14.537801 13564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef248707f044554812345a8122761ea" candidate_uuid: "fd6890f7a16e4769a342b54b037eca8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75e2228c379b4351b4697d509a6fdbc4" is_pre_election: true
I20260620 00:09:14.537930 13564 raft_consensus.cc:2454] T 4ef248707f044554812345a8122761ea P 75e2228c379b4351b4697d509a6fdbc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd6890f7a16e4769a342b54b037eca8b in term 0.
I20260620 00:09:14.538066 13366 leader_election.cc:302] T 4ef248707f044554812345a8122761ea P fd6890f7a16e4769a342b54b037eca8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b29d70e53db24fdf9a0524ea2f97241f, fd6890f7a16e4769a342b54b037eca8b; no voters:
I20260620 00:09:14.538231 13649 raft_consensus.cc:2781] T 4ef248707f044554812345a8122761ea P fd6890f7a16e4769a342b54b037eca8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:09:14.538287 13649 raft_consensus.cc:482] T 4ef248707f044554812345a8122761ea P fd6890f7a16e4769a342b54b037eca8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:09:14.538311 13649 raft_consensus.cc:3037] T 4ef248707f044554812345a8122761ea P fd6890f7a16e4769a342b54b037eca8b [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:09:14.539229 13649 raft_consensus.cc:504] T 4ef248707f044554812345a8122761ea P fd6890f7a16e4769a342b54b037eca8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd6890f7a16e4769a342b54b037eca8b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36535 } } peers { permanent_uuid: "75e2228c379b4351b4697d509a6fdbc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36743 } } peers { permanent_uuid: "b29d70e53db24fdf9a0524ea2f97241f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41743 } }
I20260620 00:09:14.539371 13649 leader_election.cc:288] T 4ef248707f044554812345a8122761ea P fd6890f7a16e4769a342b54b037eca8b [CANDIDATE]: Term 1 election: Requested vote from peers 75e2228c379b4351b4697d509a6fdbc4 (127.0.98.3:36743), b29d70e53db24fdf9a0524ea2f97241f (127.0.98.1:41743)
I20260620 00:09:14.539479 13564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef248707f044554812345a8122761ea" candidate_uuid: "fd6890f7a16e4769a342b54b037eca8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75e2228c379b4351b4697d509a6fdbc4"
I20260620 00:09:14.539547 13564 raft_consensus.cc:3037] T 4ef248707f044554812345a8122761ea P 75e2228c379b4351b4697d509a6fdbc4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:09:14.539712 13277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef248707f044554812345a8122761ea" candidate_uuid: "fd6890f7a16e4769a342b54b037eca8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b29d70e53db24fdf9a0524ea2f97241f"
I20260620 00:09:14.539784 13277 raft_consensus.cc:3037] T 4ef248707f044554812345a8122761ea P b29d70e53db24fdf9a0524ea2f97241f [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:09:14.540412 13564 raft_consensus.cc:2454] T 4ef248707f044554812345a8122761ea P 75e2228c379b4351b4697d509a6fdbc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd6890f7a16e4769a342b54b037eca8b in term 1.
I20260620 00:09:14.540591 13277 raft_consensus.cc:2454] T 4ef248707f044554812345a8122761ea P b29d70e53db24fdf9a0524ea2f97241f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd6890f7a16e4769a342b54b037eca8b in term 1.
I20260620 00:09:14.540668 13363 leader_election.cc:302] T 4ef248707f044554812345a8122761ea P fd6890f7a16e4769a342b54b037eca8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75e2228c379b4351b4697d509a6fdbc4, fd6890f7a16e4769a342b54b037eca8b; no voters:
I20260620 00:09:14.540817 13649 raft_consensus.cc:2781] T 4ef248707f044554812345a8122761ea P fd6890f7a16e4769a342b54b037eca8b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:09:14.540997 13649 raft_consensus.cc:686] T 4ef248707f044554812345a8122761ea P fd6890f7a16e4769a342b54b037eca8b [term 1 LEADER]: Becoming Leader. State: Replica: fd6890f7a16e4769a342b54b037eca8b, State: Running, Role: LEADER
I20260620 00:09:14.541100 13649 consensus_queue.cc:237] T 4ef248707f044554812345a8122761ea P fd6890f7a16e4769a342b54b037eca8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd6890f7a16e4769a342b54b037eca8b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36535 } } peers { permanent_uuid: "75e2228c379b4351b4697d509a6fdbc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36743 } } peers { permanent_uuid: "b29d70e53db24fdf9a0524ea2f97241f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41743 } }
I20260620 00:09:14.542523 13144 catalog_manager.cc:5495] T 4ef248707f044554812345a8122761ea P fd6890f7a16e4769a342b54b037eca8b reported cstate change: term changed from 0 to 1, leader changed from <none> to fd6890f7a16e4769a342b54b037eca8b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "fd6890f7a16e4769a342b54b037eca8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd6890f7a16e4769a342b54b037eca8b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75e2228c379b4351b4697d509a6fdbc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b29d70e53db24fdf9a0524ea2f97241f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41743 } health_report { overall_health: UNKNOWN } } }
I20260620 00:09:14.563372 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:14.560054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44672 (local address 127.0.98.2:36535)
0620 00:09:14.560104 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:14.560106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:14.560117 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:09:14.560205 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:14.560206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:14.560229 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:14.560312 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:09:14.560477 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:14.561246 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:14.562400 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:14.562693 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:14.562780 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:14.562982 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:09:14.563120 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:09:14.563209 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:14.563267 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 00:09:14.563291 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 00:09:14.565596 13564 raft_consensus.cc:1261] T 4ef248707f044554812345a8122761ea P 75e2228c379b4351b4697d509a6fdbc4 [term 1 FOLLOWER]: Refusing update from remote peer fd6890f7a16e4769a342b54b037eca8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:09:14.565935 13277 raft_consensus.cc:1261] T 4ef248707f044554812345a8122761ea P b29d70e53db24fdf9a0524ea2f97241f [term 1 FOLLOWER]: Refusing update from remote peer fd6890f7a16e4769a342b54b037eca8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:09:14.566270 13654 consensus_queue.cc:1035] T 4ef248707f044554812345a8122761ea P fd6890f7a16e4769a342b54b037eca8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "75e2228c379b4351b4697d509a6fdbc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:09:14.566524 13654 consensus_queue.cc:1035] T 4ef248707f044554812345a8122761ea P fd6890f7a16e4769a342b54b037eca8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29d70e53db24fdf9a0524ea2f97241f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:09:14.573873 13657 mvcc.cc:205] Tried to move back new op lower bound from 7298720377097494528 to 7298720377001009152. Current Snapshot: MvccSnapshot[applied={T|T < 7298720377097494528}]
I20260620 00:09:14.574987 13660 mvcc.cc:205] Tried to move back new op lower bound from 7298720377097494528 to 7298720377001009152. Current Snapshot: MvccSnapshot[applied={T|T < 7298720377097494528}]
I20260620 00:09:14.576311 13658 mvcc.cc:205] Tried to move back new op lower bound from 7298720377097494528 to 7298720377001009152. Current Snapshot: MvccSnapshot[applied={T|T < 7298720377097494528}]
I20260620 00:09:14.579314 13143 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:09:14.579420 13143 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:09:14.580250 13143 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ebfd1638ec78413c8a1c9b23d16bf956: Sending DeleteTablet for 3 replicas of tablet 4ef248707f044554812345a8122761ea
I20260620 00:09:14.580801 13410 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ef248707f044554812345a8122761ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:09:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44664
I20260620 00:09:14.581027 13528 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ef248707f044554812345a8122761ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:09:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39322
I20260620 00:09:14.581254 13268 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ef248707f044554812345a8122761ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:09:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54814
I20260620 00:09:14.582046 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 13182
I20260620 00:09:14.583940 13667 tablet_replica.cc:330] T 4ef248707f044554812345a8122761ea P 75e2228c379b4351b4697d509a6fdbc4: stopping tablet replica
I20260620 00:09:14.584125 13666 tablet_replica.cc:330] T 4ef248707f044554812345a8122761ea P fd6890f7a16e4769a342b54b037eca8b: stopping tablet replica
I20260620 00:09:14.584300 13666 raft_consensus.cc:2229] T 4ef248707f044554812345a8122761ea P fd6890f7a16e4769a342b54b037eca8b [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:09:14.584446 13667 raft_consensus.cc:2229] T 4ef248707f044554812345a8122761ea P 75e2228c379b4351b4697d509a6fdbc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:09:14.584632 13666 raft_consensus.cc:2258] T 4ef248707f044554812345a8122761ea P fd6890f7a16e4769a342b54b037eca8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:09:14.584825 13667 raft_consensus.cc:2258] T 4ef248707f044554812345a8122761ea P 75e2228c379b4351b4697d509a6fdbc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:09:14.585743 13667 ts_tablet_manager.cc:1905] T 4ef248707f044554812345a8122761ea P 75e2228c379b4351b4697d509a6fdbc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:09:14.586018 13666 ts_tablet_manager.cc:1905] T 4ef248707f044554812345a8122761ea P fd6890f7a16e4769a342b54b037eca8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:09:14.589361 13667 ts_tablet_manager.cc:1918] T 4ef248707f044554812345a8122761ea P 75e2228c379b4351b4697d509a6fdbc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:09:14.589601 13666 ts_tablet_manager.cc:1918] T 4ef248707f044554812345a8122761ea P fd6890f7a16e4769a342b54b037eca8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:09:14.589745 13666 log.cc:1196] T 4ef248707f044554812345a8122761ea P fd6890f7a16e4769a342b54b037eca8b: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-1/wal/wals/4ef248707f044554812345a8122761ea
W20260620 00:09:14.589972 13131 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41743 (error 108)
I20260620 00:09:14.590215 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 13346
I20260620 00:09:14.591799 13667 log.cc:1196] T 4ef248707f044554812345a8122761ea P 75e2228c379b4351b4697d509a6fdbc4: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914001714546-392-0/minicluster-data/ts-2/wal/wals/4ef248707f044554812345a8122761ea
I20260620 00:09:14.592234 13667 ts_tablet_manager.cc:1939] T 4ef248707f044554812345a8122761ea P 75e2228c379b4351b4697d509a6fdbc4: Deleting consensus metadata
I20260620 00:09:14.593044 13128 catalog_manager.cc:4841] TS 75e2228c379b4351b4697d509a6fdbc4 (127.0.98.3:36743): tablet 4ef248707f044554812345a8122761ea (table test-table [id=41018e443bb949d3b762c7399fef77df]) successfully deleted
I20260620 00:09:14.593631 13637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:14.593508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41743 (local address 127.0.0.1:54826)
0620 00:09:14.593555 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:14.593596 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260620 00:09:14.593924 13131 catalog_manager.cc:4584] TS b29d70e53db24fdf9a0524ea2f97241f (127.0.98.1:41743): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4ef248707f044554812345a8122761ea: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41743: connect: Connection refused (error 111)
I20260620 00:09:14.600925 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 13478
I20260620 00:09:14.605688 13637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:14.605544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36535 (local address 127.0.0.1:44680)
0620 00:09:14.605611 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:14.605642 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 00:09:14.608166 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 13111
I20260620 00:09:14.613389 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 00:09:14.958141 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 00:09:14.958242 392 mini_ranger.cc:66] Stopping Ranger...
I20260620 00:09:15.303226 392 mini_ranger.cc:68] Stopped Ranger
2026-06-20 00:09:15.303 UTC [12178] LOG: received smart shutdown request
2026-06-20 00:09:15.307 UTC [12178] LOG: background worker "logical replication launcher" (PID 12187) exited with exit code 1
2026-06-20 00:09:15.307 UTC [12182] LOG: shutting down
2026-06-20 00:09:15.501 UTC [12178] LOG: database system is shut down
2026-06-20T00:09:15Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50300 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:09:15 dist-test-slave-grqt krb5kdc[13678](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:09:15 dist-test-slave-grqt krb5kdc[13678](info): set up 2 sockets
Jun 20 00:09:15 dist-test-slave-grqt krb5kdc[13678](info): commencing operation
krb5kdc: starting...
W20260620 00:09:16.653347 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:09:16 dist-test-slave-grqt krb5kdc[13678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:09:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:09:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:09:16.773877 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34149
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34149,127.0.98.61:35257,127.0.98.60:39079
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:09:16.854965 13691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:09:16.855237 13691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:09:16.855280 13691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:09:16.856489 13691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:09:16.856567 13691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:09:16.856592 13691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:09:16.856614 13691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:09:16.856643 13691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:09:16.858261 13691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34149,127.0.98.61:35257,127.0.98.60:39079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:09:16.858553 13691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:09:16.858800 13691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:09:16.864081 13696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:16.864094 13699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:09:16.864167 13691 server_base.cc:1034] running on GCE node
W20260620 00:09:16.864094 13697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:09:16.864598 13691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:09:16.864939 13691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:09:16.866117 13691 hybrid_clock.cc:648] HybridClock initialized: now 1781914156866074 us; error 58 us; skew 500 ppm
Jun 20 00:09:16 dist-test-slave-grqt krb5kdc[13678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:09:16.868953 13691 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:09:16.869452 13691 webserver.cc:460] Webserver started at http://127.0.98.62:46247/ using document root <none> and password file <none>
I20260620 00:09:16.869716 13691 fs_manager.cc:359] Metadata directory not provided
I20260620 00:09:16.869793 13691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:09:16.869947 13691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:09:16.871101 13691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "3e5cd3f72e97409581e3457df6c650d9"
format_stamp: "Formatted at 2026-06-20 00:09:16 on dist-test-slave-grqt"
server_key: "5595825713fafc39a480d2ff8bfefd3c"
server_key_iv: "6517291cd5334c302d79317145513677"
server_key_version: "encryptionkey@0"
I20260620 00:09:16.871493 13691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "3e5cd3f72e97409581e3457df6c650d9"
format_stamp: "Formatted at 2026-06-20 00:09:16 on dist-test-slave-grqt"
server_key: "5595825713fafc39a480d2ff8bfefd3c"
server_key_iv: "6517291cd5334c302d79317145513677"
server_key_version: "encryptionkey@0"
I20260620 00:09:16.872960 13691 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:09:16.873903 13706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:16.874148 13691 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:09:16.874269 13691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "3e5cd3f72e97409581e3457df6c650d9"
format_stamp: "Formatted at 2026-06-20 00:09:16 on dist-test-slave-grqt"
server_key: "5595825713fafc39a480d2ff8bfefd3c"
server_key_iv: "6517291cd5334c302d79317145513677"
server_key_version: "encryptionkey@0"
I20260620 00:09:16.874365 13691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:09:16.883870 13691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:09:16.886453 13691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:09:16.886641 13691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:09:16.891188 13691 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34149
I20260620 00:09:16.891223 13758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34149 every 8 connection(s)
I20260620 00:09:16.891623 13691 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:09:16.892697 13759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:09:16.894477 13759 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34149 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:09:16.898665 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 13691
I20260620 00:09:16.898763 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:09:16.898984 392 external_mini_cluster.cc:1428] Setting key 7fbfa87d39d0d6138eaaf8d5a1d4d716
Jun 20 00:09:16 dist-test-slave-grqt krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:09:16.905969 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:16.895308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34149 (local address 127.0.0.1:58502)
0620 00:09:16.895682 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:16.895714 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:16.896200 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:16.896518 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:16.896520 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:16.896786 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:09:16.896946 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:16.896950 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:16.898104 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:16.898105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:16.898945 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:16.898948 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:16.899482 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:16.899483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:16.899546 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:16.900185 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:09:16.900205 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:09:16.901970 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:09:16.904323 (+ 2353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:16.904330 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:16.904332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:16.904539 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:16.904848 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:16.904848 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:16.904849 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:16.904931 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:16.905460 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:09:16.905466 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:09:16.905530 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 00:09:16.905618 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 00:09:16.905766 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":119,"threads_started":1}
I20260620 00:09:16.905994 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:16.895392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58502 (local address 127.0.98.62:34149)
0620 00:09:16.895624 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:16.895632 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:16.895815 (+ 183us) server_negotiation.cc:407] Connection header received
0620 00:09:16.896367 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:16.896369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:16.896388 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:16.896475 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:09:16.897057 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:16.897993 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:16.899048 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:16.899378 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:16.899514 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:16.902068 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:09:16.902076 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:09:16.902076 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:09:16.902084 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:09:16.904203 (+ 2119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:16.904649 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:16.904653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:16.904654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:16.904771 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:16.905001 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:16.905002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:16.905002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:16.905320 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:09:16.905449 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:16.905668 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 00:09:16.905838 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":189,"thread_start_us":83,"threads_started":1}
I20260620 00:09:16.906903 13759 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35257 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:09:16.907315 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:16.907165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35257 (local address 127.0.0.1:33002)
0620 00:09:16.907223 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:16.907277 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 00:09:16.907492 13708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35257: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260620 00:09:16.907986 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:16.907895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35257 (local address 127.0.0.1:33012)
0620 00:09:16.907929 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:16.907960 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260620 00:09:16.908105 13759 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35257: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35257: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 00:09:16.929286 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35257
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34149,127.0.98.61:35257,127.0.98.60:39079
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:09:16.957356 13759 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35257 } attempt: 1
I20260620 00:09:16.957935 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:16.957798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35257 (local address 127.0.0.1:33022)
0620 00:09:16.957862 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:16.957907 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 00:09:16.958287 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:16.958189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35257 (local address 127.0.0.1:33032)
0620 00:09:16.958236 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:16.958262 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 00:09:16.958395 13759 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35257: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35257: connect: Connection refused (error 111)
W20260620 00:09:17.014223 13766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:09:17.014448 13766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:09:17.014474 13766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:09:17.015704 13766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:09:17.015758 13766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:09:17.015774 13766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:09:17.015786 13766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:09:17.015799 13766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:09:17.017449 13766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34149,127.0.98.61:35257,127.0.98.60:39079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:09:17.017730 13766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:09:17.017992 13766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:09:17.022997 13772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:17.023008 13771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:17.022997 13774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:09:17.023603 13766 server_base.cc:1034] running on GCE node
I20260620 00:09:17.023901 13766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:09:17.024226 13766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:09:17.025454 13766 hybrid_clock.cc:648] HybridClock initialized: now 1781914157025440 us; error 58 us; skew 500 ppm
I20260620 00:09:17.026619 13759 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35257 } attempt: 2
I20260620 00:09:17.027064 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:17.026904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35257 (local address 127.0.0.1:33038)
0620 00:09:17.026983 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:17.027022 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 00:09:17.027468 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:17.027342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35257 (local address 127.0.0.1:33040)
0620 00:09:17.027414 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:17.027445 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 00:09:17.027630 13759 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35257: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35257: connect: Connection refused (error 111)
Jun 20 00:09:17 dist-test-slave-grqt krb5kdc[13678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:09:17.028589 13766 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 00:09:17.029117 13766 webserver.cc:460] Webserver started at http://127.0.98.61:34865/ using document root <none> and password file <none>
I20260620 00:09:17.029412 13766 fs_manager.cc:359] Metadata directory not provided
I20260620 00:09:17.029477 13766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:09:17.029618 13766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:09:17.030761 13766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-1/data/instance:
uuid: "fb0e5789a2d7426bbf8b9a73a1f3de62"
format_stamp: "Formatted at 2026-06-20 00:09:17 on dist-test-slave-grqt"
server_key: "0072897b3e390a6845394551521673bf"
server_key_iv: "c61c501acfac592a3be3ceb8bc47f398"
server_key_version: "encryptionkey@0"
I20260620 00:09:17.031194 13766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-1/wal/instance:
uuid: "fb0e5789a2d7426bbf8b9a73a1f3de62"
format_stamp: "Formatted at 2026-06-20 00:09:17 on dist-test-slave-grqt"
server_key: "0072897b3e390a6845394551521673bf"
server_key_iv: "c61c501acfac592a3be3ceb8bc47f398"
server_key_version: "encryptionkey@0"
I20260620 00:09:17.032611 13766 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:09:17.033492 13781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:17.033733 13766 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:09:17.033844 13766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-1/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-1/wal
uuid: "fb0e5789a2d7426bbf8b9a73a1f3de62"
format_stamp: "Formatted at 2026-06-20 00:09:17 on dist-test-slave-grqt"
server_key: "0072897b3e390a6845394551521673bf"
server_key_iv: "c61c501acfac592a3be3ceb8bc47f398"
server_key_version: "encryptionkey@0"
I20260620 00:09:17.033933 13766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:09:17.060534 13766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:09:17.062924 13766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:09:17.063112 13766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:09:17.067390 13766 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:35257
I20260620 00:09:17.067421 13833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:35257 every 8 connection(s)
I20260620 00:09:17.067782 13766 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-1/data/info.pb
I20260620 00:09:17.068686 13834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:09:17.070406 13834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34149 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:09:17.075163 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 13766
I20260620 00:09:17.075310 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-1/wal/instance
I20260620 00:09:17.075534 392 external_mini_cluster.cc:1428] Setting key 2a58a351141320426f136f7b783c5995
Jun 20 00:09:17 dist-test-slave-grqt krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:09:17.081029 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:17.071198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58512 (local address 127.0.98.62:34149)
0620 00:09:17.071346 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:17.071348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:17.071509 (+ 161us) server_negotiation.cc:407] Connection header received
0620 00:09:17.072043 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:17.072044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:17.072060 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:17.072106 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:09:17.072761 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.073278 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.074331 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.074543 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.074619 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:17.077128 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:09:17.077130 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:09:17.077130 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:09:17.077140 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:09:17.079321 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:17.079853 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:17.079855 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:17.079855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:17.079924 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:17.080176 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:17.080178 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:17.080178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:17.080349 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:09:17.080432 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:17.080751 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 00:09:17.080883 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 00:09:17.081058 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:17.071121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34149 (local address 127.0.0.1:58512)
0620 00:09:17.071357 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:17.071414 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:17.071920 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:17.072195 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:17.072199 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:17.072465 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:09:17.072658 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:17.072662 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.073416 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.073417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:17.074221 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:17.074223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.074637 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.074638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:17.074769 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:17.075315 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:09:17.075326 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:09:17.077020 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:09:17.079436 (+ 2416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:17.079447 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:17.079450 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:17.079761 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:17.079999 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:17.080000 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:17.080000 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:17.080106 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:17.080443 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:09:17.080450 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:09:17.080602 (+ 152us) client_negotiation.cc:769] Sending connection context
0620 00:09:17.080702 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 00:09:17.080877 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":84,"threads_started":1}
I20260620 00:09:17.081739 13834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35257 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 20 00:09:17 dist-test-slave-grqt krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 00:09:17.090715 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:17.082064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35257 (local address 127.0.0.1:33048)
0620 00:09:17.082108 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:17.082118 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:17.082159 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:17.082392 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:17.082393 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:17.082562 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:09:17.082630 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:17.082633 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.083546 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.083547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:17.084424 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:17.084426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.084994 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.084995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:17.085070 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:17.085585 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:09:17.085589 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:09:17.087599 (+ 2010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:09:17.089601 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:17.089602 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:17.089602 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:17.089761 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:17.089988 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:17.089988 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:17.089989 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:17.090056 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:17.090378 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:09:17.090379 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:09:17.090417 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 00:09:17.090490 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 00:09:17.090595 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 00:09:17.090801 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:17.082119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33048 (local address 127.0.98.61:35257)
0620 00:09:17.082260 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:17.082264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:17.082279 (+ 15us) server_negotiation.cc:407] Connection header received
0620 00:09:17.082295 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:17.082297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:17.082307 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:17.082359 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:09:17.082722 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.083458 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.084518 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.084861 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.084982 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:17.087693 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:09:17.087701 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:09:17.087701 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:09:17.087708 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:09:17.089501 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:17.089858 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:17.089861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:17.089861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:17.089922 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:17.090120 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:17.090121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:17.090121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:17.090306 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:09:17.090403 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:17.090521 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 00:09:17.090666 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
I20260620 00:09:17.091418 13834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39079 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:09:17.091773 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:17.091651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39079 (local address 127.0.0.1:44474)
0620 00:09:17.091699 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:17.091747 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 00:09:17.091912 13785 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39079: connect: Connection refused (error 111)
I20260620 00:09:17.092345 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:17.092246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39079 (local address 127.0.0.1:44488)
0620 00:09:17.092293 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:17.092323 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 00:09:17.092445 13834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39079: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39079: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 00:09:17.105721 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39079
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34149,127.0.98.61:35257,127.0.98.60:39079
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:09:17.118810 13759 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35257 } attempt: 3
Jun 20 00:09:17 dist-test-slave-grqt krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 00:09:17.126235 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:17.119261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33052 (local address 127.0.98.61:35257)
0620 00:09:17.119315 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:17.119318 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:17.119392 (+ 74us) server_negotiation.cc:407] Connection header received
0620 00:09:17.119463 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:17.119464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:17.119483 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:17.119553 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:09:17.119885 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.120394 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.121105 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.121331 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.121487 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:17.123579 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:09:17.123581 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:09:17.123581 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:09:17.123588 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:09:17.125013 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:17.125419 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:17.125420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:17.125420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:17.125481 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:17.125681 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:17.125682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:17.125682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:17.125803 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:09:17.125887 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:17.126037 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 00:09:17.126134 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 00:09:17.126235 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:17.119261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35257 (local address 127.0.0.1:33052)
0620 00:09:17.119322 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:17.119333 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:17.119394 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:17.119563 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:17.119564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:17.119731 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:09:17.119797 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:17.119799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.120479 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.120481 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:17.121005 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:17.121006 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.121489 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.121490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:17.121583 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:17.122147 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:09:17.122152 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:09:17.123459 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:09:17.125130 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:17.125132 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:17.125132 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:17.125312 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:17.125550 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:17.125551 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:17.125551 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:17.125614 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:17.125872 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:09:17.125873 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:09:17.125921 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 00:09:17.126011 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 00:09:17.126119 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 00:09:17.126655 13759 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39079 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:09:17.127048 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:17.126912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39079 (local address 127.0.0.1:44504)
0620 00:09:17.126972 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:17.127015 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 00:09:17.127419 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:17.127316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39079 (local address 127.0.0.1:44518)
0620 00:09:17.127372 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:17.127400 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 00:09:17.127571 13759 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39079: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39079: connect: Connection refused (error 111)
I20260620 00:09:17.141670 13834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39079 } attempt: 1
I20260620 00:09:17.142167 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:17.142031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39079 (local address 127.0.0.1:44524)
0620 00:09:17.142093 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:17.142135 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 00:09:17.142558 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:17.142454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39079 (local address 127.0.0.1:44532)
0620 00:09:17.142505 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:17.142533 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 00:09:17.142647 13834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39079: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39079: connect: Connection refused (error 111)
I20260620 00:09:17.158737 13759 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39079 } attempt: 1
I20260620 00:09:17.159243 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:17.159074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39079 (local address 127.0.0.1:44534)
0620 00:09:17.159143 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:17.159195 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 00:09:17.159627 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:17.159523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39079 (local address 127.0.0.1:44550)
0620 00:09:17.159573 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:17.159605 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 00:09:17.159729 13759 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39079: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39079: connect: Connection refused (error 111)
W20260620 00:09:17.198757 13841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:09:17.199007 13841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:09:17.199031 13841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:09:17.200320 13841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:09:17.200376 13841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:09:17.200390 13841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:09:17.200403 13841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:09:17.200416 13841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:09:17.202149 13841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34149,127.0.98.61:35257,127.0.98.60:39079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:09:17.202476 13841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:09:17.202760 13841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:09:17.208096 13849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:17.208098 13846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:17.208097 13847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:09:17.208434 13841 server_base.cc:1034] running on GCE node
I20260620 00:09:17.208781 13841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:09:17.209146 13841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:09:17.210366 13841 hybrid_clock.cc:648] HybridClock initialized: now 1781914157210353 us; error 74 us; skew 500 ppm
I20260620 00:09:17.210783 13834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39079 } attempt: 2
I20260620 00:09:17.211210 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:17.211074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39079 (local address 127.0.0.1:44552)
0620 00:09:17.211138 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:17.211171 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 00:09:17.211720 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:17.211474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39079 (local address 127.0.0.1:44566)
0620 00:09:17.211670 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:17.211695 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":85,"threads_started":1}
W20260620 00:09:17.211843 13834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39079: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39079: connect: Connection refused (error 111)
Jun 20 00:09:17 dist-test-slave-grqt krb5kdc[13678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:09:17.213531 13841 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 00:09:17.213975 13841 webserver.cc:460] Webserver started at http://127.0.98.60:36557/ using document root <none> and password file <none>
I20260620 00:09:17.214175 13841 fs_manager.cc:359] Metadata directory not provided
I20260620 00:09:17.214221 13841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:09:17.214315 13841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:09:17.215395 13841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-2/data/instance:
uuid: "feb7c279c93048c0b9c29bf83cb7d336"
format_stamp: "Formatted at 2026-06-20 00:09:17 on dist-test-slave-grqt"
server_key: "b21218a25122d4581e8342b14ff8023a"
server_key_iv: "da823eb136f52f416a56fc26d580543f"
server_key_version: "encryptionkey@0"
I20260620 00:09:17.215732 13841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-2/wal/instance:
uuid: "feb7c279c93048c0b9c29bf83cb7d336"
format_stamp: "Formatted at 2026-06-20 00:09:17 on dist-test-slave-grqt"
server_key: "b21218a25122d4581e8342b14ff8023a"
server_key_iv: "da823eb136f52f416a56fc26d580543f"
server_key_version: "encryptionkey@0"
I20260620 00:09:17.217124 13841 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:09:17.218101 13857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:17.218305 13841 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:09:17.218401 13841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-2/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-2/wal
uuid: "feb7c279c93048c0b9c29bf83cb7d336"
format_stamp: "Formatted at 2026-06-20 00:09:17 on dist-test-slave-grqt"
server_key: "b21218a25122d4581e8342b14ff8023a"
server_key_iv: "da823eb136f52f416a56fc26d580543f"
server_key_version: "encryptionkey@0"
I20260620 00:09:17.218465 13841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:09:17.234906 13759 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39079 } attempt: 2
I20260620 00:09:17.235399 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:17.235257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39079 (local address 127.0.0.1:44572)
0620 00:09:17.235326 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:17.235367 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 00:09:17.235796 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:17.235695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39079 (local address 127.0.0.1:44582)
0620 00:09:17.235747 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:17.235772 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 00:09:17.235909 13759 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39079: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39079: connect: Connection refused (error 111)
I20260620 00:09:17.236698 13841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:09:17.239251 13841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:09:17.239423 13841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:09:17.243721 13841 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39079
I20260620 00:09:17.243767 13909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39079 every 8 connection(s)
I20260620 00:09:17.244138 13841 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-2/data/info.pb
I20260620 00:09:17.245043 13910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:09:17.246675 13910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34149 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:09:17.251312 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 13841
I20260620 00:09:17.251446 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914001714546-392-0/minicluster-data/master-2/wal/instance
I20260620 00:09:17.251689 392 external_mini_cluster.cc:1428] Setting key 983832887b08fe7234a9689b65d22810
Jun 20 00:09:17 dist-test-slave-grqt krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:09:17.256234 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:17.247700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58524 (local address 127.0.98.62:34149)
0620 00:09:17.247769 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:17.247771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:17.247781 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:09:17.248109 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:17.248110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:17.248118 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:17.248186 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:09:17.248799 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.249262 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.250256 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.250447 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.250514 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:17.253127 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:09:17.253129 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:09:17.253130 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:09:17.253139 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:09:17.254735 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:17.255238 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:17.255239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:17.255239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:17.255298 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:17.255588 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:17.255589 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:17.255590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:17.255711 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:09:17.255790 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:17.256059 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 00:09:17.256143 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 00:09:17.256435 13912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:17.247363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34149 (local address 127.0.0.1:58524)
0620 00:09:17.247533 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:17.247576 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:17.248025 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:17.248242 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:17.248245 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:17.248525 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:09:17.248724 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:17.248728 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.249340 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.249340 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:17.250160 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:17.250163 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.250529 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.250529 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:17.250631 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:17.251303 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:09:17.251312 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:09:17.253033 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:09:17.254837 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:17.254844 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:17.254846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:17.255127 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:17.255370 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:17.255371 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:17.255371 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:17.255515 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:17.255781 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:09:17.255787 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:09:17.255928 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 00:09:17.256038 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 00:09:17.256183 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":99,"threads_started":1}
I20260620 00:09:17.257138 13910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35257 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 00:09:17 dist-test-slave-grqt krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 00:09:17.261437 13914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:17.252244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44588 (local address 127.0.98.60:39079)
0620 00:09:17.252424 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:17.252431 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:17.252453 (+ 22us) server_negotiation.cc:407] Connection header received
0620 00:09:17.252472 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:17.252474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:17.252491 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:17.252580 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:09:17.253285 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.254070 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.254902 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.255352 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.255424 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:17.258285 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:09:17.258292 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:09:17.258292 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:09:17.258299 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:09:17.260058 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:17.260506 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:17.260509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:17.260509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:17.260563 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:17.260794 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:17.260795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:17.260795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:17.260924 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:09:17.261074 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:17.261201 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 00:09:17.261275 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":93,"threads_started":1}
Jun 20 00:09:17 dist-test-slave-grqt krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 00:09:17.264662 13912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:17.257490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35257 (local address 127.0.0.1:33062)
0620 00:09:17.257536 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:17.257546 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:17.257584 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:17.257708 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:17.257709 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:17.257870 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:09:17.257931 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:17.257933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.258766 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.258767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:17.259496 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:17.259498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.259982 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.259983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:17.260042 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:17.260470 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:09:17.260476 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:09:17.261947 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:09:17.263648 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:17.263649 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:17.263649 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:17.263815 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:17.264028 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:17.264029 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:17.264029 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:17.264085 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:17.264342 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:09:17.264343 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:09:17.264392 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 00:09:17.264470 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 00:09:17.264563 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 00:09:17.264662 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:17.257548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33062 (local address 127.0.98.61:35257)
0620 00:09:17.257586 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:17.257588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:17.257596 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:09:17.257649 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:17.257650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:17.257658 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:17.257703 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:09:17.258005 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.258676 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.259593 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.259897 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.259980 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:17.262093 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:09:17.262094 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:09:17.262095 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:09:17.262103 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:09:17.263549 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:17.263900 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:17.263901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:17.263901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:17.263955 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:17.264153 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:17.264154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:17.264154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:17.264270 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:09:17.264340 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:17.264496 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 00:09:17.264567 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 00:09:17.265053 13910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39079 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 00:09:17 dist-test-slave-grqt krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 00:09:17 dist-test-slave-grqt krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 00:09:17.270252 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:17.262345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58526 (local address 127.0.98.62:34149)
0620 00:09:17.262409 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:17.262411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:17.262420 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:09:17.262447 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:17.262448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:17.262458 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:17.262515 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:09:17.263209 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.263730 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.264469 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.264699 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.264789 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:17.267273 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:09:17.267275 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:09:17.267276 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:09:17.267283 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:09:17.268969 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:17.269456 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:17.269457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:17.269457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:17.269513 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:17.269720 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:17.269721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:17.269721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:17.269820 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:09:17.270016 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:17.270053 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 00:09:17.270135 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 00:09:17.272763 13912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:17.265307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39079 (local address 127.0.0.1:44590)
0620 00:09:17.265425 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:17.265435 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:17.265479 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:17.265596 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:17.265597 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:17.265730 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:09:17.265755 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:17.265756 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.266864 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.266864 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:17.267410 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:17.267412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.267945 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.267946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:17.268011 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:17.268401 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:09:17.268406 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:09:17.269894 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:09:17.271745 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:17.271746 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:17.271747 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:17.271947 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:17.272133 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:17.272134 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:17.272134 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:17.272183 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:17.272427 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:09:17.272428 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:09:17.272459 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 00:09:17.272504 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 00:09:17.272578 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260620 00:09:17.272763 13914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:17.265359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44590 (local address 127.0.98.60:39079)
0620 00:09:17.265426 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:17.265428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:17.265464 (+ 36us) server_negotiation.cc:407] Connection header received
0620 00:09:17.265531 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:17.265532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:17.265542 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:17.265594 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:09:17.265826 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.266275 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.267516 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.267867 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.267919 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:17.269978 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:09:17.269979 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:09:17.269980 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:09:17.269987 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:09:17.271642 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:17.272011 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:17.272012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:17.272012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:17.272069 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:17.272247 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:17.272248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:17.272248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:17.272365 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:09:17.272427 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:17.272542 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 00:09:17.272640 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":50}
I20260620 00:09:17.274590 13910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P feb7c279c93048c0b9c29bf83cb7d336: Bootstrap starting.
I20260620 00:09:17.275460 13910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P feb7c279c93048c0b9c29bf83cb7d336: Neither blocks nor log segments found. Creating new log.
Jun 20 00:09:17 dist-test-slave-grqt krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 00:09:17.275805 13910 log.cc:824] T 00000000000000000000000000000000 P feb7c279c93048c0b9c29bf83cb7d336: Log is configured to *not* fsync() on all Append() calls
I20260620 00:09:17.276806 13910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P feb7c279c93048c0b9c29bf83cb7d336: No bootstrap required, opened a new log
I20260620 00:09:17.278915 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:17.271181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33076 (local address 127.0.98.61:35257)
0620 00:09:17.271239 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:17.271240 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:17.271246 (+ 6us) server_negotiation.cc:407] Connection header received
0620 00:09:17.271265 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:17.271266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:17.271274 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:17.271337 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:09:17.271851 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.272559 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.273469 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.273746 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.273821 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:17.276202 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:09:17.276203 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:09:17.276204 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:09:17.276212 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:09:17.277750 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:17.278179 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:17.278180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:17.278180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:17.278240 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:17.278440 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:17.278441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:17.278441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:17.278541 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:09:17.278650 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:17.278764 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 00:09:17.278830 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 00:09:17.278965 13910 raft_consensus.cc:348] T 00000000000000000000000000000000 P feb7c279c93048c0b9c29bf83cb7d336 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e5cd3f72e97409581e3457df6c650d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34149 } } peers { permanent_uuid: "fb0e5789a2d7426bbf8b9a73a1f3de62" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35257 } } peers { permanent_uuid: "feb7c279c93048c0b9c29bf83cb7d336" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39079 } }
I20260620 00:09:17.279145 13910 raft_consensus.cc:374] T 00000000000000000000000000000000 P feb7c279c93048c0b9c29bf83cb7d336 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:09:17.279191 13910 raft_consensus.cc:729] T 00000000000000000000000000000000 P feb7c279c93048c0b9c29bf83cb7d336 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb7c279c93048c0b9c29bf83cb7d336, State: Initialized, Role: FOLLOWER
I20260620 00:09:17.279322 13910 consensus_queue.cc:260] T 00000000000000000000000000000000 P feb7c279c93048c0b9c29bf83cb7d336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e5cd3f72e97409581e3457df6c650d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34149 } } peers { permanent_uuid: "fb0e5789a2d7426bbf8b9a73a1f3de62" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35257 } } peers { permanent_uuid: "feb7c279c93048c0b9c29bf83cb7d336" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39079 } }
I20260620 00:09:17.279646 13910 sys_catalog.cc:564] T 00000000000000000000000000000000 P feb7c279c93048c0b9c29bf83cb7d336 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:09:17.279624 13916 sys_catalog.cc:455] T 00000000000000000000000000000000 P feb7c279c93048c0b9c29bf83cb7d336 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e5cd3f72e97409581e3457df6c650d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34149 } } peers { permanent_uuid: "fb0e5789a2d7426bbf8b9a73a1f3de62" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35257 } } peers { permanent_uuid: "feb7c279c93048c0b9c29bf83cb7d336" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39079 } } }
I20260620 00:09:17.279752 13916 sys_catalog.cc:458] T 00000000000000000000000000000000 P feb7c279c93048c0b9c29bf83cb7d336 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:09:17.281031 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260620 00:09:17.281842 13927 catalog_manager.cc:1558] T 00000000000000000000000000000000 P feb7c279c93048c0b9c29bf83cb7d336: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:09:17.281971 13927 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:09:17.281862 13841 master_runner.cc:186] Error getting master registration for 127.0.98.62:34149: OK, instance_id { permanent_uuid: "3e5cd3f72e97409581e3457df6c650d9" instance_seqno: 1781914156889795 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 00:09:17.284001 13841 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 00:09:17.294303 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:17.285983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58534 (local address 127.0.98.62:34149)
0620 00:09:17.286037 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:17.286040 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:17.286176 (+ 136us) server_negotiation.cc:407] Connection header received
0620 00:09:17.286321 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:17.286323 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:17.286333 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:17.286365 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:09:17.286985 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.287761 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.288888 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.289186 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.289243 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:17.291011 (+ 1768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:09:17.291013 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:09:17.291013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:09:17.291021 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:09:17.292800 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:17.293201 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:17.293203 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:17.293203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:17.293272 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:17.293508 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:17.293509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:17.293510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:17.293651 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:09:17.293768 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:17.294104 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 00:09:17.294183 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 00:09:17.294704 13914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:17.286154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44602 (local address 127.0.98.60:39079)
0620 00:09:17.286203 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:17.286205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:17.287090 (+ 885us) server_negotiation.cc:407] Connection header received
0620 00:09:17.287178 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:17.287179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:17.287188 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:17.287217 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:09:17.287906 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.288632 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.289605 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.289877 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.289942 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:17.291756 (+ 1814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:09:17.291757 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:09:17.291757 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:09:17.291765 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:09:17.293401 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:17.293772 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:17.293774 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:17.293774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:17.293832 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:17.294046 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:17.294046 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:17.294047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:17.294217 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:09:17.294301 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:17.294532 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 00:09:17.294607 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 00:09:17.295491 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:17.285763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33090 (local address 127.0.98.61:35257)
0620 00:09:17.285807 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:17.285809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:17.285951 (+ 142us) server_negotiation.cc:407] Connection header received
0620 00:09:17.286064 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:17.286065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:17.286077 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:17.286146 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:09:17.286945 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.287686 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.288930 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.289222 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.289267 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:17.291813 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:09:17.291814 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:09:17.291815 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:09:17.291823 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:09:17.294076 (+ 2253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:17.294459 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:17.294460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:17.294460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:17.294514 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:17.294706 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:17.294707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:17.294707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:17.294879 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:09:17.294960 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:17.295311 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 00:09:17.295404 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 00:09:17.302999 13834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39079 } attempt: 3
Jun 20 00:09:17 dist-test-slave-grqt krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 00:09:17.310979 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:17.303320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39079 (local address 127.0.0.1:44616)
0620 00:09:17.303415 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:17.303429 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:17.303490 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:17.303626 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:17.303627 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:17.303812 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:09:17.303898 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:17.303900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.304708 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.304709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:17.305598 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:17.305601 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.305965 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.305966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:17.306040 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:17.306494 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:09:17.306499 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:09:17.307920 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:09:17.309878 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:17.309879 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:17.309879 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:17.310059 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:17.310290 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:17.310291 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:17.310292 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:17.310366 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:17.310675 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:09:17.310676 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:09:17.310722 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 00:09:17.310787 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 00:09:17.310896 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 00:09:17.311031 13914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:17.303391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44616 (local address 127.0.98.60:39079)
0620 00:09:17.303438 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:17.303440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:17.303473 (+ 33us) server_negotiation.cc:407] Connection header received
0620 00:09:17.303555 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:17.303556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:17.303566 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:17.303620 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:09:17.303991 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.304629 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.305699 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.305889 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.305947 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:17.308003 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:09:17.308004 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:09:17.308005 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:09:17.308012 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:09:17.309734 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:17.310136 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:17.310137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:17.310137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:17.310201 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:17.310442 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:17.310442 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:17.310443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:17.310583 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:09:17.310653 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:17.310830 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 00:09:17.310919 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 00:09:17.312714 13834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb0e5789a2d7426bbf8b9a73a1f3de62: Bootstrap starting.
I20260620 00:09:17.313539 13834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb0e5789a2d7426bbf8b9a73a1f3de62: Neither blocks nor log segments found. Creating new log.
I20260620 00:09:17.313889 13834 log.cc:824] T 00000000000000000000000000000000 P fb0e5789a2d7426bbf8b9a73a1f3de62: Log is configured to *not* fsync() on all Append() calls
I20260620 00:09:17.314802 13834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb0e5789a2d7426bbf8b9a73a1f3de62: No bootstrap required, opened a new log
I20260620 00:09:17.316776 13834 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb0e5789a2d7426bbf8b9a73a1f3de62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e5cd3f72e97409581e3457df6c650d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34149 } } peers { permanent_uuid: "fb0e5789a2d7426bbf8b9a73a1f3de62" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35257 } } peers { permanent_uuid: "feb7c279c93048c0b9c29bf83cb7d336" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39079 } }
I20260620 00:09:17.316939 13834 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb0e5789a2d7426bbf8b9a73a1f3de62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:09:17.316977 13834 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb0e5789a2d7426bbf8b9a73a1f3de62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb0e5789a2d7426bbf8b9a73a1f3de62, State: Initialized, Role: FOLLOWER
I20260620 00:09:17.317097 13834 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb0e5789a2d7426bbf8b9a73a1f3de62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e5cd3f72e97409581e3457df6c650d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34149 } } peers { permanent_uuid: "fb0e5789a2d7426bbf8b9a73a1f3de62" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35257 } } peers { permanent_uuid: "feb7c279c93048c0b9c29bf83cb7d336" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39079 } }
I20260620 00:09:17.317461 13834 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb0e5789a2d7426bbf8b9a73a1f3de62 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:09:17.317441 13937 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb0e5789a2d7426bbf8b9a73a1f3de62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e5cd3f72e97409581e3457df6c650d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34149 } } peers { permanent_uuid: "fb0e5789a2d7426bbf8b9a73a1f3de62" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35257 } } peers { permanent_uuid: "feb7c279c93048c0b9c29bf83cb7d336" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39079 } } }
I20260620 00:09:17.317605 13937 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb0e5789a2d7426bbf8b9a73a1f3de62 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 00:09:17.319785 13948 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fb0e5789a2d7426bbf8b9a73a1f3de62: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:09:17.319918 13948 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:09:17.319846 13766 master_runner.cc:186] Error getting master registration for 127.0.98.62:34149: OK, instance_id { permanent_uuid: "3e5cd3f72e97409581e3457df6c650d9" instance_seqno: 1781914156889795 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 00:09:17.322225 13766 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 00:09:17.335071 13759 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39079 } attempt: 3
Jun 20 00:09:17 dist-test-slave-grqt krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 00:09:17.343644 13914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:17.335455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44620 (local address 127.0.98.60:39079)
0620 00:09:17.335498 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:17.335501 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:17.335536 (+ 35us) server_negotiation.cc:407] Connection header received
0620 00:09:17.335561 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:17.335561 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:17.335571 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:17.335626 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:09:17.335998 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.336764 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.337893 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.338217 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.338308 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:17.340585 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:09:17.340586 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:09:17.340587 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:09:17.340595 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:09:17.342370 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:17.342774 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:17.342775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:17.342775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:17.342839 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:17.343063 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:17.343064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:17.343064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:17.343216 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:09:17.343287 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:17.343455 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 00:09:17.343537 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 00:09:17.343657 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:17.335392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39079 (local address 127.0.0.1:44620)
0620 00:09:17.335444 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:17.335455 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:17.335494 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:17.335647 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:17.335648 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:17.335848 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:09:17.335907 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:17.335910 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.336854 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.336855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:17.337787 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:17.337790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:17.338312 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:17.338313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:17.338392 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:17.338910 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:09:17.338915 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:09:17.340487 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:09:17.342464 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:17.342465 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:17.342466 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:17.342692 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:17.342916 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:17.342917 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:17.342917 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:17.342982 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:17.343289 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:09:17.343290 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:09:17.343343 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 00:09:17.343417 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 00:09:17.343544 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 00:09:17.345783 13759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e5cd3f72e97409581e3457df6c650d9: Bootstrap starting.
I20260620 00:09:17.346539 13759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e5cd3f72e97409581e3457df6c650d9: Neither blocks nor log segments found. Creating new log.
I20260620 00:09:17.346841 13759 log.cc:824] T 00000000000000000000000000000000 P 3e5cd3f72e97409581e3457df6c650d9: Log is configured to *not* fsync() on all Append() calls
I20260620 00:09:17.347581 13937 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb0e5789a2d7426bbf8b9a73a1f3de62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:09:17.347676 13937 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb0e5789a2d7426bbf8b9a73a1f3de62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e5cd3f72e97409581e3457df6c650d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34149 } } peers { permanent_uuid: "fb0e5789a2d7426bbf8b9a73a1f3de62" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35257 } } peers { permanent_uuid: "feb7c279c93048c0b9c29bf83cb7d336" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39079 } }
I20260620 00:09:17.347942 13937 leader_election.cc:288] T 00000000000000000000000000000000 P fb0e5789a2d7426bbf8b9a73a1f3de62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e5cd3f72e97409581e3457df6c650d9 (127.0.98.62:34149), feb7c279c93048c0b9c29bf83cb7d336 (127.0.98.60:39079)
I20260620 00:09:17.348160 13759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e5cd3f72e97409581e3457df6c650d9: No bootstrap required, opened a new log
I20260620 00:09:17.348242 13884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb0e5789a2d7426bbf8b9a73a1f3de62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb7c279c93048c0b9c29bf83cb7d336" is_pre_election: true
I20260620 00:09:17.348383 13884 raft_consensus.cc:2454] T 00000000000000000000000000000000 P feb7c279c93048c0b9c29bf83cb7d336 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb0e5789a2d7426bbf8b9a73a1f3de62 in term 0.
I20260620 00:09:17.348573 13785 leader_election.cc:302] T 00000000000000000000000000000000 P fb0e5789a2d7426bbf8b9a73a1f3de62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: fb0e5789a2d7426bbf8b9a73a1f3de62, feb7c279c93048c0b9c29bf83cb7d336; no voters:
I20260620 00:09:17.348817 13937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb0e5789a2d7426bbf8b9a73a1f3de62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:09:17.348869 13937 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb0e5789a2d7426bbf8b9a73a1f3de62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:09:17.348908 13937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb0e5789a2d7426bbf8b9a73a1f3de62 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:09:17.349776 13937 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb0e5789a2d7426bbf8b9a73a1f3de62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e5cd3f72e97409581e3457df6c650d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34149 } } peers { permanent_uuid: "fb0e5789a2d7426bbf8b9a73a1f3de62" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35257 } } peers { permanent_uuid: "feb7c279c93048c0b9c29bf83cb7d336" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39079 } }
I20260620 00:09:17.349901 13937 leader_election.cc:288] T 00000000000000000000000000000000 P fb0e5789a2d7426bbf8b9a73a1f3de62 [CANDIDATE]: Term 1 election: Requested vote from peers 3e5cd3f72e97409581e3457df6c650d9 (127.0.98.62:34149), feb7c279c93048c0b9c29bf83cb7d336 (127.0.98.60:39079)
I20260620 00:09:17.350241 13884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb0e5789a2d7426bbf8b9a73a1f3de62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb7c279c93048c0b9c29bf83cb7d336"
I20260620 00:09:17.350317 13884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P feb7c279c93048c0b9c29bf83cb7d336 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:09:17.350478 13759 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e5cd3f72e97409581e3457df6c650d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e5cd3f72e97409581e3457df6c650d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34149 } } peers { permanent_uuid: "fb0e5789a2d7426bbf8b9a73a1f3de62" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35257 } } peers { permanent_uuid: "feb7c279c93048c0b9c29bf83cb7d336" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39079 } }
I20260620 00:09:17.350664 13759 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e5cd3f72e97409581e3457df6c650d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:09:17.350731 13759 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e5cd3f72e97409581e3457df6c650d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e5cd3f72e97409581e3457df6c650d9, State: Initialized, Role: FOLLOWER
I20260620 00:09:17.350919 13759 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e5cd3f72e97409581e3457df6c650d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e5cd3f72e97409581e3457df6c650d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34149 } } peers { permanent_uuid: "fb0e5789a2d7426bbf8b9a73a1f3de62" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35257 } } peers { permanent_uuid: "feb7c279c93048c0b9c29bf83cb7d336" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39079 } }
I20260620 00:09:17.351171 13884 raft_consensus.cc:2454] T 00000000000000000000000000000000 P feb7c279c93048c0b9c29bf83cb7d336 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb0e5789a2d7426bbf8b9a73a1f3de62 in term 1.
I20260620 00:09:17.351320 13785 leader_election.cc:302] T 00000000000000000000000000000000 P fb0e5789a2d7426bbf8b9a73a1f3de62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: fb0e5789a2d7426bbf8b9a73a1f3de62, feb7c279c93048c0b9c29bf83cb7d336; no voters:
I20260620 00:09:17.351428 13732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb0e5789a2d7426bbf8b9a73a1f3de62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e5cd3f72e97409581e3457df6c650d9"
I20260620 00:09:17.351521 13937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb0e5789a2d7426bbf8b9a73a1f3de62 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:09:17.351540 13732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e5cd3f72e97409581e3457df6c650d9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:09:17.351658 13937 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb0e5789a2d7426bbf8b9a73a1f3de62 [term 1 LEADER]: Becoming Leader. State: Replica: fb0e5789a2d7426bbf8b9a73a1f3de62, State: Running, Role: LEADER
I20260620 00:09:17.351751 13937 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb0e5789a2d7426bbf8b9a73a1f3de62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e5cd3f72e97409581e3457df6c650d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34149 } } peers { permanent_uuid: "fb0e5789a2d7426bbf8b9a73a1f3de62" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35257 } } peers { permanent_uuid: "feb7c279c93048c0b9c29bf83cb7d336" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39079 } }
I20260620 00:09:17.352265 13937 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb0e5789a2d7426bbf8b9a73a1f3de62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb0e5789a2d7426bbf8b9a73a1f3de62. Latest consensus state: current_term: 1 leader_uuid: "fb0e5789a2d7426bbf8b9a73a1f3de62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e5cd3f72e97409581e3457df6c650d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34149 } } peers { permanent_uuid: "fb0e5789a2d7426bbf8b9a73a1f3de62" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35257 } } peers { permanent_uuid: "feb7c279c93048c0b9c29bf83cb7d336" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39079 } } }
I20260620 00:09:17.352339 13937 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb0e5789a2d7426bbf8b9a73a1f3de62 [sys.catalog]: This master's current role is: LEADER
I20260620 00:09:17.352623 13954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:09:17.352754 13732 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3e5cd3f72e97409581e3457df6c650d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb0e5789a2d7426bbf8b9a73a1f3de62 in term 1.
I20260620 00:09:17.352847 13954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:09:17.351432 13733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb0e5789a2d7426bbf8b9a73a1f3de62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e5cd3f72e97409581e3457df6c650d9" is_pre_election: true
I20260620 00:09:17.353118 13733 raft_consensus.cc:2362] T 00000000000000000000000000000000 P 3e5cd3f72e97409581e3457df6c650d9 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate fb0e5789a2d7426bbf8b9a73a1f3de62 in term 1. Re-sending same reply.
I20260620 00:09:17.351377 13759 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e5cd3f72e97409581e3457df6c650d9 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 00:09:17.355144 13966 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3e5cd3f72e97409581e3457df6c650d9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:09:17.355252 13966 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:09:17.361498 13951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e5cd3f72e97409581e3457df6c650d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e5cd3f72e97409581e3457df6c650d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34149 } } peers { permanent_uuid: "fb0e5789a2d7426bbf8b9a73a1f3de62" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35257 } } peers { permanent_uuid: "feb7c279c93048c0b9c29bf83cb7d336" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39079 } } }
I20260620 00:09:17.361613 13951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e5cd3f72e97409581e3457df6c650d9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:09:17.362382 13884 raft_consensus.cc:1261] T 00000000000000000000000000000000 P feb7c279c93048c0b9c29bf83cb7d336 [term 1 FOLLOWER]: Refusing update from remote peer fb0e5789a2d7426bbf8b9a73a1f3de62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:09:17.362773 13952 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fb0e5789a2d7426bbf8b9a73a1f3de62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb7c279c93048c0b9c29bf83cb7d336" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:09:17.364141 13916 sys_catalog.cc:455] T 00000000000000000000000000000000 P feb7c279c93048c0b9c29bf83cb7d336 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb0e5789a2d7426bbf8b9a73a1f3de62. Latest consensus state: current_term: 1 leader_uuid: "fb0e5789a2d7426bbf8b9a73a1f3de62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e5cd3f72e97409581e3457df6c650d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34149 } } peers { permanent_uuid: "fb0e5789a2d7426bbf8b9a73a1f3de62" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35257 } } peers { permanent_uuid: "feb7c279c93048c0b9c29bf83cb7d336" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39079 } } }
I20260620 00:09:17.364228 13916 sys_catalog.cc:458] T 00000000000000000000000000000000 P feb7c279c93048c0b9c29bf83cb7d336 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:09:17.364673 13733 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3e5cd3f72e97409581e3457df6c650d9 [term 1 FOLLOWER]: Refusing update from remote peer fb0e5789a2d7426bbf8b9a73a1f3de62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:09:17.365007 13952 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fb0e5789a2d7426bbf8b9a73a1f3de62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e5cd3f72e97409581e3457df6c650d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:09:17.366276 13951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e5cd3f72e97409581e3457df6c650d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb0e5789a2d7426bbf8b9a73a1f3de62. Latest consensus state: current_term: 1 leader_uuid: "fb0e5789a2d7426bbf8b9a73a1f3de62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e5cd3f72e97409581e3457df6c650d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34149 } } peers { permanent_uuid: "fb0e5789a2d7426bbf8b9a73a1f3de62" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35257 } } peers { permanent_uuid: "feb7c279c93048c0b9c29bf83cb7d336" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39079 } } }
I20260620 00:09:17.366366 13951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e5cd3f72e97409581e3457df6c650d9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:09:17.367165 13952 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb0e5789a2d7426bbf8b9a73a1f3de62 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fb0e5789a2d7426bbf8b9a73a1f3de62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e5cd3f72e97409581e3457df6c650d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34149 } } peers { permanent_uuid: "fb0e5789a2d7426bbf8b9a73a1f3de62" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35257 } } peers { permanent_uuid: "feb7c279c93048c0b9c29bf83cb7d336" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39079 } } }
I20260620 00:09:17.367249 13952 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb0e5789a2d7426bbf8b9a73a1f3de62 [sys.catalog]: This master's current role is: LEADER
I20260620 00:09:17.367663 13916 sys_catalog.cc:455] T 00000000000000000000000000000000 P feb7c279c93048c0b9c29bf83cb7d336 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fb0e5789a2d7426bbf8b9a73a1f3de62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e5cd3f72e97409581e3457df6c650d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34149 } } peers { permanent_uuid: "fb0e5789a2d7426bbf8b9a73a1f3de62" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35257 } } peers { permanent_uuid: "feb7c279c93048c0b9c29bf83cb7d336" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39079 } } }
I20260620 00:09:17.367740 13916 sys_catalog.cc:458] T 00000000000000000000000000000000 P feb7c279c93048c0b9c29bf83cb7d336 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:09:17.368211 13955 mvcc.cc:205] Tried to move back new op lower bound from 7298720388553633792 to 7298720388513906688. Current Snapshot: MvccSnapshot[applied={T|T < 7298720388553633792}]
I20260620 00:09:17.368784 13954 catalog_manager.cc:1347] Generated new cluster ID: 884d9c90fb894e0295a206a6ed21a5c1
I20260620 00:09:17.368826 13954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:09:17.373817 13968 mvcc.cc:205] Tried to move back new op lower bound from 7298720388553633792 to 7298720388513906688. Current Snapshot: MvccSnapshot[applied={T|T < 7298720388553633792}]
I20260620 00:09:17.374976 13937 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb0e5789a2d7426bbf8b9a73a1f3de62 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fb0e5789a2d7426bbf8b9a73a1f3de62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e5cd3f72e97409581e3457df6c650d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34149 } } peers { permanent_uuid: "fb0e5789a2d7426bbf8b9a73a1f3de62" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35257 } } peers { permanent_uuid: "feb7c279c93048c0b9c29bf83cb7d336" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39079 } } }
I20260620 00:09:17.375078 13937 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb0e5789a2d7426bbf8b9a73a1f3de62 [sys.catalog]: This master's current role is: LEADER
I20260620 00:09:17.375571 13951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e5cd3f72e97409581e3457df6c650d9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fb0e5789a2d7426bbf8b9a73a1f3de62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e5cd3f72e97409581e3457df6c650d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34149 } } peers { permanent_uuid: "fb0e5789a2d7426bbf8b9a73a1f3de62" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35257 } } peers { permanent_uuid: "feb7c279c93048c0b9c29bf83cb7d336" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39079 } } }
I20260620 00:09:17.375643 13951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e5cd3f72e97409581e3457df6c650d9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:09:17.375901 13970 mvcc.cc:205] Tried to move back new op lower bound from 7298720388553633792 to 7298720388513906688. Current Snapshot: MvccSnapshot[applied={T|T < 7298720388553633792}]
I20260620 00:09:17.391980 13954 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:09:17.392812 13954 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:09:17.400228 13954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb0e5789a2d7426bbf8b9a73a1f3de62: Generated new TSK 0
I20260620 00:09:17.400522 13954 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:09:18.282465 13927 catalog_manager.cc:1259] Loaded cluster ID: 884d9c90fb894e0295a206a6ed21a5c1
I20260620 00:09:18.282543 13927 catalog_manager.cc:1552] T 00000000000000000000000000000000 P feb7c279c93048c0b9c29bf83cb7d336: loading cluster ID for follower catalog manager: success
I20260620 00:09:18.283452 13927 catalog_manager.cc:1574] T 00000000000000000000000000000000 P feb7c279c93048c0b9c29bf83cb7d336: acquiring CA information for follower catalog manager: success
I20260620 00:09:18.283716 13927 catalog_manager.cc:1602] T 00000000000000000000000000000000 P feb7c279c93048c0b9c29bf83cb7d336: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 00:09:18.355734 13966 catalog_manager.cc:1259] Loaded cluster ID: 884d9c90fb894e0295a206a6ed21a5c1
I20260620 00:09:18.355814 13966 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3e5cd3f72e97409581e3457df6c650d9: loading cluster ID for follower catalog manager: success
I20260620 00:09:18.356660 13966 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3e5cd3f72e97409581e3457df6c650d9: acquiring CA information for follower catalog manager: success
I20260620 00:09:18.357002 13966 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3e5cd3f72e97409581e3457df6c650d9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 00:09:18.468679 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 13691
I20260620 00:09:18.473776 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 13766
I20260620 00:09:18.478559 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 13841
2026-06-20T00:09:18Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2868 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:09:18 dist-test-slave-grqt krb5kdc[13983](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:09:18 dist-test-slave-grqt krb5kdc[13983](info): set up 2 sockets
Jun 20 00:09:18 dist-test-slave-grqt krb5kdc[13983](info): commencing operation
krb5kdc: starting...
W20260620 00:09:19.517546 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:09:19 dist-test-slave-grqt krb5kdc[13983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:09:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:09:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:09:19.644388 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32789
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32789
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:09:19.727399 13996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:09:19.727627 13996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:09:19.727650 13996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:09:19.728802 13996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:09:19.728842 13996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:09:19.728856 13996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:09:19.728869 13996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:09:19.728886 13996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:09:19.730371 13996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:09:19.730579 13996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:09:19.730754 13996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:09:19.735992 14001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:09:19.736215 13996 server_base.cc:1034] running on GCE node
W20260620 00:09:19.735987 14004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:19.736007 14002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:09:19.736573 13996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:09:19.736922 13996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:09:19.738099 13996 hybrid_clock.cc:648] HybridClock initialized: now 1781914159738076 us; error 44 us; skew 500 ppm
Jun 20 00:09:19 dist-test-slave-grqt krb5kdc[13983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914159, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:09:19.740825 13996 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:09:19.741358 13996 webserver.cc:460] Webserver started at http://127.0.98.62:35179/ using document root <none> and password file <none>
I20260620 00:09:19.741664 13996 fs_manager.cc:359] Metadata directory not provided
I20260620 00:09:19.741745 13996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:09:19.741900 13996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:09:19.742996 13996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "2266f604c9cd4acab7e6060a8045954d"
format_stamp: "Formatted at 2026-06-20 00:09:19 on dist-test-slave-grqt"
server_key: "a1cd83e01326c6e31f8dfd5b280ce9f5"
server_key_iv: "a4ad2490531a66532a82dcc722834ac8"
server_key_version: "encryptionkey@0"
I20260620 00:09:19.743363 13996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "2266f604c9cd4acab7e6060a8045954d"
format_stamp: "Formatted at 2026-06-20 00:09:19 on dist-test-slave-grqt"
server_key: "a1cd83e01326c6e31f8dfd5b280ce9f5"
server_key_iv: "a4ad2490531a66532a82dcc722834ac8"
server_key_version: "encryptionkey@0"
I20260620 00:09:19.744711 13996 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:09:19.745486 14011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:19.745656 13996 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 00:09:19.745775 13996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "2266f604c9cd4acab7e6060a8045954d"
format_stamp: "Formatted at 2026-06-20 00:09:19 on dist-test-slave-grqt"
server_key: "a1cd83e01326c6e31f8dfd5b280ce9f5"
server_key_iv: "a4ad2490531a66532a82dcc722834ac8"
server_key_version: "encryptionkey@0"
I20260620 00:09:19.745858 13996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:09:19.759711 13996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:09:19.762154 13996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:09:19.762338 13996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:09:19.766676 13996 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32789
I20260620 00:09:19.766727 14063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32789 every 8 connection(s)
I20260620 00:09:19.767115 13996 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:09:19.767997 14064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:09:19.769210 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 13996
I20260620 00:09:19.769316 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:09:19.769568 392 external_mini_cluster.cc:1428] Setting key 8be7a9ca390cecc935a7d7710226c3df
I20260620 00:09:19.770937 14064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2266f604c9cd4acab7e6060a8045954d: Bootstrap starting.
I20260620 00:09:19.771543 14064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2266f604c9cd4acab7e6060a8045954d: Neither blocks nor log segments found. Creating new log.
I20260620 00:09:19.771847 14064 log.cc:824] T 00000000000000000000000000000000 P 2266f604c9cd4acab7e6060a8045954d: Log is configured to *not* fsync() on all Append() calls
I20260620 00:09:19.772549 14064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2266f604c9cd4acab7e6060a8045954d: No bootstrap required, opened a new log
I20260620 00:09:19.774380 14064 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2266f604c9cd4acab7e6060a8045954d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2266f604c9cd4acab7e6060a8045954d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32789 } }
I20260620 00:09:19.774535 14064 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2266f604c9cd4acab7e6060a8045954d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:09:19.774581 14064 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2266f604c9cd4acab7e6060a8045954d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2266f604c9cd4acab7e6060a8045954d, State: Initialized, Role: FOLLOWER
I20260620 00:09:19.774711 14064 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2266f604c9cd4acab7e6060a8045954d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2266f604c9cd4acab7e6060a8045954d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32789 } }
I20260620 00:09:19.774792 14064 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2266f604c9cd4acab7e6060a8045954d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:09:19.774837 14064 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2266f604c9cd4acab7e6060a8045954d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:09:19.774885 14064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2266f604c9cd4acab7e6060a8045954d [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:09:19.775418 14064 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2266f604c9cd4acab7e6060a8045954d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2266f604c9cd4acab7e6060a8045954d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32789 } }
I20260620 00:09:19.775530 14064 leader_election.cc:302] T 00000000000000000000000000000000 P 2266f604c9cd4acab7e6060a8045954d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2266f604c9cd4acab7e6060a8045954d; no voters:
I20260620 00:09:19.775720 14064 leader_election.cc:288] T 00000000000000000000000000000000 P 2266f604c9cd4acab7e6060a8045954d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:09:19.775749 14069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2266f604c9cd4acab7e6060a8045954d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:09:19.775871 14069 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2266f604c9cd4acab7e6060a8045954d [term 1 LEADER]: Becoming Leader. State: Replica: 2266f604c9cd4acab7e6060a8045954d, State: Running, Role: LEADER
I20260620 00:09:19.775972 14069 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2266f604c9cd4acab7e6060a8045954d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2266f604c9cd4acab7e6060a8045954d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32789 } }
I20260620 00:09:19.776007 14064 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2266f604c9cd4acab7e6060a8045954d [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 00:09:19 dist-test-slave-grqt krb5kdc[13983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:09:19.776301 14069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2266f604c9cd4acab7e6060a8045954d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2266f604c9cd4acab7e6060a8045954d. Latest consensus state: current_term: 1 leader_uuid: "2266f604c9cd4acab7e6060a8045954d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2266f604c9cd4acab7e6060a8045954d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32789 } } }
I20260620 00:09:19.776367 14069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2266f604c9cd4acab7e6060a8045954d [sys.catalog]: This master's current role is: LEADER
I20260620 00:09:19.776873 14070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2266f604c9cd4acab7e6060a8045954d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2266f604c9cd4acab7e6060a8045954d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2266f604c9cd4acab7e6060a8045954d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32789 } } }
I20260620 00:09:19.777012 14070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2266f604c9cd4acab7e6060a8045954d [sys.catalog]: This master's current role is: LEADER
I20260620 00:09:19.777058 14074 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:09:19.777614 14074 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:09:19.779222 14074 catalog_manager.cc:1347] Generated new cluster ID: 2dfcc243381a47458c356ef8c17a22a5
I20260620 00:09:19.779279 14074 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:09:19.781457 14067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:19.770063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36952 (local address 127.0.98.62:32789)
0620 00:09:19.770254 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:19.770265 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:19.770299 (+ 34us) server_negotiation.cc:407] Connection header received
0620 00:09:19.770841 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:19.770849 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:19.770909 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:19.771058 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:09:19.771696 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:19.772719 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:19.773547 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:19.773963 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:19.774093 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:19.777472 (+ 3379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:09:19.777476 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:09:19.777477 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:09:19.777490 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:09:19.779661 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:19.780142 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:19.780144 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:19.780145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:19.780255 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:19.780480 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:19.780481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:19.780481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:19.780777 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:09:19.780900 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:19.781075 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 00:09:19.781217 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":85,"threads_started":1}
I20260620 00:09:19.782390 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:09:19.791509 14074 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:09:19.792060 14074 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:09:19.795651 14074 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2266f604c9cd4acab7e6060a8045954d: Generated new TSK 0
I20260620 00:09:19.795851 14074 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:09:19.836413 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 13996
2026-06-20T00:09:19Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1355 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:09:19 dist-test-slave-grqt krb5kdc[14091](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:09:19 dist-test-slave-grqt krb5kdc[14091](info): set up 2 sockets
Jun 20 00:09:19 dist-test-slave-grqt krb5kdc[14091](info): commencing operation
krb5kdc: starting...
W20260620 00:09:20.865819 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:09:20 dist-test-slave-grqt krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:09:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:09:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:09:20.993839 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39857
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:09:21.077466 14104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:09:21.077692 14104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:09:21.077716 14104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:09:21.078966 14104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:09:21.079012 14104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:09:21.079027 14104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:09:21.079039 14104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:09:21.079052 14104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:09:21.080646 14104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:09:21.080909 14104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:09:21.081161 14104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:09:21.086344 14110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:21.086403 14109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:21.086344 14112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:09:21.086916 14104 server_base.cc:1034] running on GCE node
I20260620 00:09:21.087188 14104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:09:21.087518 14104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:09:21.088747 14104 hybrid_clock.cc:648] HybridClock initialized: now 1781914161088732 us; error 47 us; skew 500 ppm
Jun 20 00:09:21 dist-test-slave-grqt krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:09:21.091425 14104 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:09:21.091925 14104 webserver.cc:460] Webserver started at http://127.0.98.62:41495/ using document root <none> and password file <none>
I20260620 00:09:21.092170 14104 fs_manager.cc:359] Metadata directory not provided
I20260620 00:09:21.092248 14104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:09:21.092409 14104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:09:21.093562 14104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "97eabdebe29c4620b9981c6b4e3592e9"
format_stamp: "Formatted at 2026-06-20 00:09:21 on dist-test-slave-grqt"
server_key: "00e0c4144e4770b9adba18c0a6558286"
server_key_iv: "64768d8a24eb34a7d8cd44a62787f941"
server_key_version: "encryptionkey@0"
I20260620 00:09:21.093936 14104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "97eabdebe29c4620b9981c6b4e3592e9"
format_stamp: "Formatted at 2026-06-20 00:09:21 on dist-test-slave-grqt"
server_key: "00e0c4144e4770b9adba18c0a6558286"
server_key_iv: "64768d8a24eb34a7d8cd44a62787f941"
server_key_version: "encryptionkey@0"
I20260620 00:09:21.095305 14104 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:09:21.096081 14119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:21.096364 14104 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:09:21.096467 14104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "97eabdebe29c4620b9981c6b4e3592e9"
format_stamp: "Formatted at 2026-06-20 00:09:21 on dist-test-slave-grqt"
server_key: "00e0c4144e4770b9adba18c0a6558286"
server_key_iv: "64768d8a24eb34a7d8cd44a62787f941"
server_key_version: "encryptionkey@0"
I20260620 00:09:21.096556 14104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:09:21.109154 14104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:09:21.111213 14104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:09:21.111402 14104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:09:21.115669 14104 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39857
I20260620 00:09:21.115703 14171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39857 every 8 connection(s)
I20260620 00:09:21.116097 14104 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:09:21.117105 14172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:09:21.118822 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 14104
I20260620 00:09:21.118942 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:09:21.119163 392 external_mini_cluster.cc:1428] Setting key 2acaee3e646d5a93879032ea8c7fa8ac
I20260620 00:09:21.119957 14172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9: Bootstrap starting.
I20260620 00:09:21.120568 14172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9: Neither blocks nor log segments found. Creating new log.
I20260620 00:09:21.120918 14172 log.cc:824] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9: Log is configured to *not* fsync() on all Append() calls
I20260620 00:09:21.121707 14172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9: No bootstrap required, opened a new log
I20260620 00:09:21.123489 14172 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97eabdebe29c4620b9981c6b4e3592e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39857 } }
I20260620 00:09:21.123631 14172 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:09:21.123652 14172 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97eabdebe29c4620b9981c6b4e3592e9, State: Initialized, Role: FOLLOWER
I20260620 00:09:21.123785 14172 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97eabdebe29c4620b9981c6b4e3592e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39857 } }
I20260620 00:09:21.123875 14172 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:09:21.123907 14172 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:09:21.123991 14172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:09:21.124737 14172 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97eabdebe29c4620b9981c6b4e3592e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39857 } }
I20260620 00:09:21.124872 14172 leader_election.cc:302] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97eabdebe29c4620b9981c6b4e3592e9; no voters:
I20260620 00:09:21.125063 14172 leader_election.cc:288] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:09:21.125113 14177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:09:21.125262 14177 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [term 1 LEADER]: Becoming Leader. State: Replica: 97eabdebe29c4620b9981c6b4e3592e9, State: Running, Role: LEADER
I20260620 00:09:21.125324 14172 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:09:21.125413 14177 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97eabdebe29c4620b9981c6b4e3592e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39857 } }
I20260620 00:09:21.125813 14178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97eabdebe29c4620b9981c6b4e3592e9. Latest consensus state: current_term: 1 leader_uuid: "97eabdebe29c4620b9981c6b4e3592e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97eabdebe29c4620b9981c6b4e3592e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39857 } } }
I20260620 00:09:21.125962 14178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [sys.catalog]: This master's current role is: LEADER
Jun 20 00:09:21 dist-test-slave-grqt krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:09:21.126231 14186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:09:21.126580 14179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97eabdebe29c4620b9981c6b4e3592e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97eabdebe29c4620b9981c6b4e3592e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39857 } } }
I20260620 00:09:21.126705 14179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [sys.catalog]: This master's current role is: LEADER
I20260620 00:09:21.126734 14186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:09:21.128459 14186 catalog_manager.cc:1347] Generated new cluster ID: 4b1958d88f7b469eb7dd03fa0979b716
I20260620 00:09:21.128511 14186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:09:21.130668 14175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:21.119731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35588 (local address 127.0.98.62:39857)
0620 00:09:21.119942 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:21.119951 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:21.119977 (+ 26us) server_negotiation.cc:407] Connection header received
0620 00:09:21.120444 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:21.120450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:21.120498 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:21.120620 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:09:21.121494 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:21.122445 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:21.123225 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:21.123710 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:21.123844 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:21.126876 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:09:21.126881 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:09:21.126881 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:09:21.126895 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:09:21.129008 (+ 2113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:21.129435 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:21.129438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:21.129439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:21.129549 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:21.129761 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:21.129762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:21.129762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:21.130070 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:09:21.130219 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:21.130325 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 00:09:21.130478 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":125,"threads_started":1}
I20260620 00:09:21.152410 14186 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:09:21.153163 14186 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:09:21.164623 14186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9: Generated new TSK 0
I20260620 00:09:21.164847 14186 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:09:21.196441 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39857
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:09:21.280615 14198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:09:21.280885 14198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:09:21.280931 14198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:09:21.282205 14198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:09:21.282253 14198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:09:21.282281 14198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:09:21.283809 14198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:09:21.284088 14198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:09:21.284343 14198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:09:21.289542 14203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:21.289546 14204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:09:21.289742 14198 server_base.cc:1034] running on GCE node
W20260620 00:09:21.289567 14206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:09:21.290055 14198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:09:21.290315 14198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:09:21.291483 14198 hybrid_clock.cc:648] HybridClock initialized: now 1781914161291472 us; error 31 us; skew 500 ppm
Jun 20 00:09:21 dist-test-slave-grqt krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:09:21.294152 14198 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:09:21.294548 14198 webserver.cc:460] Webserver started at http://127.0.98.1:45377/ using document root <none> and password file <none>
I20260620 00:09:21.294728 14198 fs_manager.cc:359] Metadata directory not provided
I20260620 00:09:21.294768 14198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:09:21.294863 14198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:09:21.295872 14198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/data/instance:
uuid: "ab60a5329cbf4d7995ad7e331de3529d"
format_stamp: "Formatted at 2026-06-20 00:09:21 on dist-test-slave-grqt"
server_key: "18d71d70ac1afa913059b328807c0be7"
server_key_iv: "bc6bdfa337a99afc268ef29bfdb7eaf2"
server_key_version: "encryptionkey@0"
I20260620 00:09:21.296175 14198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ab60a5329cbf4d7995ad7e331de3529d"
format_stamp: "Formatted at 2026-06-20 00:09:21 on dist-test-slave-grqt"
server_key: "18d71d70ac1afa913059b328807c0be7"
server_key_iv: "bc6bdfa337a99afc268ef29bfdb7eaf2"
server_key_version: "encryptionkey@0"
I20260620 00:09:21.297498 14198 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 00:09:21.298233 14213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:21.298478 14198 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:09:21.298574 14198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/wal
uuid: "ab60a5329cbf4d7995ad7e331de3529d"
format_stamp: "Formatted at 2026-06-20 00:09:21 on dist-test-slave-grqt"
server_key: "18d71d70ac1afa913059b328807c0be7"
server_key_iv: "bc6bdfa337a99afc268ef29bfdb7eaf2"
server_key_version: "encryptionkey@0"
I20260620 00:09:21.298635 14198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:09:21.311407 14198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:09:21.313829 14198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:09:21.313947 14198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:09:21.314142 14198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:09:21.314469 14198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:09:21.314507 14198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:21.314532 14198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:09:21.314544 14198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:21.320232 14198 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39455
I20260620 00:09:21.320308 14326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39455 every 8 connection(s)
I20260620 00:09:21.320645 14198 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:09:21.321651 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 14198
I20260620 00:09:21.321758 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:09:21.322016 392 external_mini_cluster.cc:1428] Setting key 32fd375a8630d0bb1a739902aa5621cd
Jun 20 00:09:21 dist-test-slave-grqt krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 00:09:21.332257 14175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:21.321496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59359 (local address 127.0.98.62:39857)
0620 00:09:21.321565 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:21.321568 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:21.321780 (+ 212us) server_negotiation.cc:407] Connection header received
0620 00:09:21.322554 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:21.322555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:21.322566 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:21.322631 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:09:21.324024 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:21.324846 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:21.326035 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:21.326387 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:21.326474 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:21.329094 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:09:21.329096 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:09:21.329096 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:09:21.329105 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:09:21.330875 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:21.331347 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:21.331349 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:21.331349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:21.331416 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:21.331682 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:21.331683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:21.331683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:21.331797 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:09:21.331868 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:21.332113 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 00:09:21.332185 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 00:09:21.332495 14329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:21.321445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.1:59359)
0620 00:09:21.321623 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:21.321677 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:21.322451 (+ 774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:21.322731 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:21.322736 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:21.323628 (+ 892us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:09:21.323917 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:21.323924 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:21.324949 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:21.324950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:21.325958 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:21.325961 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:21.326480 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:21.326481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:21.326653 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:21.327302 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:09:21.327311 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:09:21.328984 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:09:21.330966 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:21.330973 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:21.330975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:21.331274 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:21.331489 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:21.331490 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:21.331490 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:21.331593 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:21.331872 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:09:21.331877 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:09:21.332000 (+ 123us) client_negotiation.cc:769] Sending connection context
0620 00:09:21.332084 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 00:09:21.332210 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":85,"threads_started":1}
I20260620 00:09:21.333024 14327 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39857
I20260620 00:09:21.333127 14327 heartbeater.cc:464] Registering TS with master...
I20260620 00:09:21.333285 14327 heartbeater.cc:511] Master 127.0.98.62:39857 requested a full tablet report, sending...
I20260620 00:09:21.333823 14136 ts_manager.cc:194] Registered new tserver with Master: ab60a5329cbf4d7995ad7e331de3529d (127.0.98.1:39455)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 00:09:21.334404 14136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59359
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:09:21.352708 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39857
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:09:21.435592 14332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:09:21.435799 14332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:09:21.435823 14332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:09:21.437001 14332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:09:21.437052 14332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:09:21.437084 14332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:09:21.438593 14332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:09:21.438854 14332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:09:21.439087 14332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:09:21.444308 14338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:21.444473 14340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:09:21.444501 14332 server_base.cc:1034] running on GCE node
W20260620 00:09:21.444614 14337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:09:21.444821 14332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:09:21.445062 14332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:09:21.446204 14332 hybrid_clock.cc:648] HybridClock initialized: now 1781914161446192 us; error 29 us; skew 500 ppm
Jun 20 00:09:21 dist-test-slave-grqt krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:09:21.448755 14332 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:09:21.449275 14332 webserver.cc:460] Webserver started at http://127.0.98.2:36799/ using document root <none> and password file <none>
I20260620 00:09:21.449549 14332 fs_manager.cc:359] Metadata directory not provided
I20260620 00:09:21.449623 14332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:09:21.449779 14332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:09:21.450862 14332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/data/instance:
uuid: "5d26247cfb494b33bd2ecbe1494d3ea9"
format_stamp: "Formatted at 2026-06-20 00:09:21 on dist-test-slave-grqt"
server_key: "57f2e713c9b24fbb66ef98a1b8cb90af"
server_key_iv: "1db291533152684dc6d9581b19a58b58"
server_key_version: "encryptionkey@0"
I20260620 00:09:21.451231 14332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5d26247cfb494b33bd2ecbe1494d3ea9"
format_stamp: "Formatted at 2026-06-20 00:09:21 on dist-test-slave-grqt"
server_key: "57f2e713c9b24fbb66ef98a1b8cb90af"
server_key_iv: "1db291533152684dc6d9581b19a58b58"
server_key_version: "encryptionkey@0"
I20260620 00:09:21.452646 14332 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:09:21.453498 14347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:21.453677 14332 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:09:21.453778 14332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/wal
uuid: "5d26247cfb494b33bd2ecbe1494d3ea9"
format_stamp: "Formatted at 2026-06-20 00:09:21 on dist-test-slave-grqt"
server_key: "57f2e713c9b24fbb66ef98a1b8cb90af"
server_key_iv: "1db291533152684dc6d9581b19a58b58"
server_key_version: "encryptionkey@0"
I20260620 00:09:21.453890 14332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:09:21.509017 14332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:09:21.511304 14332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:09:21.511489 14332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:09:21.511752 14332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:09:21.512081 14332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:09:21.512138 14332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:21.512176 14332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:09:21.512223 14332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:21.518661 14332 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38605
I20260620 00:09:21.518698 14460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38605 every 8 connection(s)
I20260620 00:09:21.519052 14332 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 00:09:21 dist-test-slave-grqt krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:09:21.528748 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 14332
I20260620 00:09:21.528918 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:09:21.529126 392 external_mini_cluster.cc:1428] Setting key 7dd8cd39e39865914cc5b28b92e1ba85
I20260620 00:09:21.529268 14175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:21.519885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56471 (local address 127.0.98.62:39857)
0620 00:09:21.519963 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:21.519966 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:21.520163 (+ 197us) server_negotiation.cc:407] Connection header received
0620 00:09:21.520749 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:21.520750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:21.520762 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:21.520817 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:09:21.521859 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:21.522362 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:21.523407 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:21.523604 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:21.523693 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:21.526169 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:09:21.526170 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:09:21.526171 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:09:21.526179 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:09:21.527732 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:21.528208 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:21.528209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:21.528209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:21.528271 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:21.528561 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:21.528562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:21.528562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:21.528681 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:09:21.528762 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:21.529080 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 00:09:21.529171 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 00:09:21.529554 14463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:21.519865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.2:56471)
0620 00:09:21.520031 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:21.520067 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:21.520636 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:21.520897 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:21.520900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:21.521535 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:09:21.521749 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:21.521755 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:21.522464 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:21.522466 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:21.523285 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:21.523287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:21.523704 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:21.523705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:21.523834 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:21.524417 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:09:21.524425 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:09:21.526066 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:09:21.527852 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:21.527861 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:21.527864 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:21.528110 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:21.528348 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:21.528348 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:21.528349 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:21.528471 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:21.528752 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:09:21.528759 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:09:21.528946 (+ 187us) client_negotiation.cc:769] Sending connection context
0620 00:09:21.529060 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 00:09:21.529230 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":66,"threads_started":1}
I20260620 00:09:21.530138 14461 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39857
I20260620 00:09:21.530261 14461 heartbeater.cc:464] Registering TS with master...
I20260620 00:09:21.530452 14461 heartbeater.cc:511] Master 127.0.98.62:39857 requested a full tablet report, sending...
I20260620 00:09:21.530861 14136 ts_manager.cc:194] Registered new tserver with Master: 5d26247cfb494b33bd2ecbe1494d3ea9 (127.0.98.2:38605)
I20260620 00:09:21.531378 14136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56471
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:09:21.556510 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39857
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:09:21.639719 14466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:09:21.639953 14466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:09:21.639981 14466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:09:21.641254 14466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:09:21.641300 14466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:09:21.641328 14466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:09:21.642853 14466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:09:21.643050 14466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:09:21.643232 14466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:09:21.648823 14471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:21.648818 14474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:21.648818 14472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:09:21.648876 14466 server_base.cc:1034] running on GCE node
I20260620 00:09:21.649224 14466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:09:21.649521 14466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:09:21.650724 14466 hybrid_clock.cc:648] HybridClock initialized: now 1781914161650687 us; error 57 us; skew 500 ppm
Jun 20 00:09:21 dist-test-slave-grqt krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:09:21.653443 14466 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:09:21.653887 14466 webserver.cc:460] Webserver started at http://127.0.98.3:39401/ using document root <none> and password file <none>
I20260620 00:09:21.654115 14466 fs_manager.cc:359] Metadata directory not provided
I20260620 00:09:21.654182 14466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:09:21.654356 14466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:09:21.655440 14466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/data/instance:
uuid: "ed930b9cba294089a7183326762ac0f9"
format_stamp: "Formatted at 2026-06-20 00:09:21 on dist-test-slave-grqt"
server_key: "f3c2ede53150f0847211bae243c9f748"
server_key_iv: "2aab5a610bd19d23eb3f821cc107fb3f"
server_key_version: "encryptionkey@0"
I20260620 00:09:21.655820 14466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ed930b9cba294089a7183326762ac0f9"
format_stamp: "Formatted at 2026-06-20 00:09:21 on dist-test-slave-grqt"
server_key: "f3c2ede53150f0847211bae243c9f748"
server_key_iv: "2aab5a610bd19d23eb3f821cc107fb3f"
server_key_version: "encryptionkey@0"
I20260620 00:09:21.657516 14466 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:09:21.658557 14481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:21.658792 14466 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:09:21.658905 14466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/wal
uuid: "ed930b9cba294089a7183326762ac0f9"
format_stamp: "Formatted at 2026-06-20 00:09:21 on dist-test-slave-grqt"
server_key: "f3c2ede53150f0847211bae243c9f748"
server_key_iv: "2aab5a610bd19d23eb3f821cc107fb3f"
server_key_version: "encryptionkey@0"
I20260620 00:09:21.659004 14466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:09:21.681460 14466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:09:21.683929 14466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:09:21.684095 14466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:09:21.684357 14466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:09:21.684675 14466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:09:21.684728 14466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:21.684775 14466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:09:21.684798 14466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:21.690860 14466 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46639
I20260620 00:09:21.690948 14594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46639 every 8 connection(s)
I20260620 00:09:21.691290 14466 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:09:21.691931 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 14466
I20260620 00:09:21.692028 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:09:21.692243 392 external_mini_cluster.cc:1428] Setting key d9e8c7cf1b7adaae583b90c869e3dd62
Jun 20 00:09:21 dist-test-slave-grqt krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:09:21.701853 14175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:21.692098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39573 (local address 127.0.98.62:39857)
0620 00:09:21.692154 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:21.692157 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:21.692443 (+ 286us) server_negotiation.cc:407] Connection header received
0620 00:09:21.693109 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:21.693110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:21.693121 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:21.693159 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:09:21.694277 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:21.694795 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:21.695814 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:21.696029 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:21.696130 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:21.698506 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:09:21.698507 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:09:21.698508 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:09:21.698516 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:09:21.700114 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:21.700697 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:21.700699 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:21.700699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:21.700763 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:21.701065 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:21.701066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:21.701066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:21.701189 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:09:21.701286 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:21.701669 (+ 383us) server_negotiation.cc:299] Negotiation successful
0620 00:09:21.701753 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 00:09:21.702064 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:21.692096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.3:39573)
0620 00:09:21.692299 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:21.692342 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:21.693000 (+ 658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:21.693253 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:21.693262 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:21.693959 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:09:21.694157 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:21.694161 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:21.694897 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:21.694898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:21.695720 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:21.695722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:21.696143 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:21.696144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:21.696287 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:21.696835 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:09:21.696843 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:09:21.698397 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:09:21.700349 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:21.700356 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:21.700359 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:21.700616 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:21.700834 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:21.700835 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:21.700836 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:21.700995 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:21.701279 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:09:21.701286 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:09:21.701513 (+ 227us) client_negotiation.cc:769] Sending connection context
0620 00:09:21.701633 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 00:09:21.701816 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":90,"threads_started":1}
I20260620 00:09:21.702613 14595 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39857
I20260620 00:09:21.702749 14595 heartbeater.cc:464] Registering TS with master...
I20260620 00:09:21.702957 14595 heartbeater.cc:511] Master 127.0.98.62:39857 requested a full tablet report, sending...
I20260620 00:09:21.703342 14136 ts_manager.cc:194] Registered new tserver with Master: ed930b9cba294089a7183326762ac0f9 (127.0.98.3:46639)
I20260620 00:09:21.703872 14136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39573
I20260620 00:09:21.703974 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:09:21.712306 14175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:21.705344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35596 (local address 127.0.98.62:39857)
0620 00:09:21.705451 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:21.705453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:21.705554 (+ 101us) server_negotiation.cc:407] Connection header received
0620 00:09:21.705644 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:21.705645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:21.705655 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:21.705714 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:09:21.706319 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:21.706795 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:21.707540 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:21.707734 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:21.707824 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:21.709249 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:09:21.709251 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:09:21.709251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:09:21.709259 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:09:21.711001 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:21.711431 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:21.711432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:21.711432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:21.711499 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:21.711703 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:21.711704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:21.711705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:21.711880 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:09:21.711995 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:21.712101 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 00:09:21.712184 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 00:09:21.714208 14136 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:09:21.715814 14136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:09:21.723276 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:21.719082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36942 (local address 127.0.98.1:39455)
0620 00:09:21.719234 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:21.719240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:21.719577 (+ 337us) server_negotiation.cc:407] Connection header received
0620 00:09:21.719683 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:21.719686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:21.719701 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:21.719799 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:09:21.720008 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:21.721084 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:21.722288 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:21.722863 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:21.722957 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:21.723022 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:21.723142 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 00:09:21.723183 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":78,"threads_started":1}
I20260620 00:09:21.723268 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:21.718856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39455 (local address 127.0.0.1:36942)
0620 00:09:21.719267 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:21.719311 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:21.719571 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:21.719890 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:21.719890 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:21.719892 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:09:21.719927 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:21.719928 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:21.721206 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:21.721206 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:21.722190 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:21.722193 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:21.722956 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:21.722957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:21.723041 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:21.723056 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 00:09:21.723128 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 00:09:21.723183 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":69,"threads_started":1}
I20260620 00:09:21.723471 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:21.719204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59856 (local address 127.0.98.2:38605)
0620 00:09:21.719380 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:21.719384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:21.719637 (+ 253us) server_negotiation.cc:407] Connection header received
0620 00:09:21.719679 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:21.719681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:21.719695 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:21.719774 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:09:21.719945 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:21.720680 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:21.722135 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:21.722922 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:21.722996 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:21.723050 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:21.723341 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 00:09:21.723393 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":106,"threads_started":1}
I20260620 00:09:21.723488 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:21.718858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46639 (local address 127.0.0.1:38454)
0620 00:09:21.719278 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:21.719311 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:21.719377 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:21.719738 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:21.719741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:21.719744 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:09:21.719816 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:21.719819 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:21.721195 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:21.721196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:21.722349 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:21.722351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:21.723266 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:21.723267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:21.723325 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:21.723328 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:09:21.723376 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 00:09:21.723424 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":55,"thread_start_us":90,"threads_started":1}
I20260620 00:09:21.723742 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:21.719014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38454 (local address 127.0.98.3:46639)
0620 00:09:21.719456 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:21.719459 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:21.719470 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:09:21.719543 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:21.719545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:21.719562 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:21.719640 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:09:21.719902 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:21.721079 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:21.722437 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:21.723123 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:21.723186 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:21.723235 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:21.723598 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 00:09:21.723647 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":119,"threads_started":1}
I20260620 00:09:21.723867 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:21.719072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38605 (local address 127.0.0.1:59856)
0620 00:09:21.719413 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:21.719422 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:21.719470 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:21.719793 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:21.719794 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:21.719796 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:09:21.719852 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:21.719855 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:21.720783 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:21.720784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:21.722036 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:21.722039 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:21.723087 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:21.723088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:21.723152 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:21.723158 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 00:09:21.723206 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 00:09:21.723258 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":74,"threads_started":1}
I20260620 00:09:21.724490 14261 tablet_service.cc:1467] Processing CreateTablet for tablet c75a9f786c2b4d0382db67fa9cba274b (DEFAULT_TABLE table=test-table [id=bbefd1009fd849c2aa4b1b124b0ba71f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:09:21.724900 14261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c75a9f786c2b4d0382db67fa9cba274b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:09:21.725037 14395 tablet_service.cc:1467] Processing CreateTablet for tablet c75a9f786c2b4d0382db67fa9cba274b (DEFAULT_TABLE table=test-table [id=bbefd1009fd849c2aa4b1b124b0ba71f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:09:21.725349 14395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c75a9f786c2b4d0382db67fa9cba274b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:09:21.727939 14612 tablet_bootstrap.cc:492] T c75a9f786c2b4d0382db67fa9cba274b P ab60a5329cbf4d7995ad7e331de3529d: Bootstrap starting.
I20260620 00:09:21.728413 14529 tablet_service.cc:1467] Processing CreateTablet for tablet c75a9f786c2b4d0382db67fa9cba274b (DEFAULT_TABLE table=test-table [id=bbefd1009fd849c2aa4b1b124b0ba71f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:09:21.728629 14612 tablet_bootstrap.cc:654] T c75a9f786c2b4d0382db67fa9cba274b P ab60a5329cbf4d7995ad7e331de3529d: Neither blocks nor log segments found. Creating new log.
I20260620 00:09:21.728761 14529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c75a9f786c2b4d0382db67fa9cba274b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:09:21.729171 14612 log.cc:824] T c75a9f786c2b4d0382db67fa9cba274b P ab60a5329cbf4d7995ad7e331de3529d: Log is configured to *not* fsync() on all Append() calls
I20260620 00:09:21.729665 14613 tablet_bootstrap.cc:492] T c75a9f786c2b4d0382db67fa9cba274b P 5d26247cfb494b33bd2ecbe1494d3ea9: Bootstrap starting.
I20260620 00:09:21.730073 14612 tablet_bootstrap.cc:492] T c75a9f786c2b4d0382db67fa9cba274b P ab60a5329cbf4d7995ad7e331de3529d: No bootstrap required, opened a new log
I20260620 00:09:21.730149 14612 ts_tablet_manager.cc:1397] T c75a9f786c2b4d0382db67fa9cba274b P ab60a5329cbf4d7995ad7e331de3529d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 00:09:21.730343 14613 tablet_bootstrap.cc:654] T c75a9f786c2b4d0382db67fa9cba274b P 5d26247cfb494b33bd2ecbe1494d3ea9: Neither blocks nor log segments found. Creating new log.
I20260620 00:09:21.730903 14613 log.cc:824] T c75a9f786c2b4d0382db67fa9cba274b P 5d26247cfb494b33bd2ecbe1494d3ea9: Log is configured to *not* fsync() on all Append() calls
I20260620 00:09:21.731700 14613 tablet_bootstrap.cc:492] T c75a9f786c2b4d0382db67fa9cba274b P 5d26247cfb494b33bd2ecbe1494d3ea9: No bootstrap required, opened a new log
I20260620 00:09:21.731767 14613 ts_tablet_manager.cc:1397] T c75a9f786c2b4d0382db67fa9cba274b P 5d26247cfb494b33bd2ecbe1494d3ea9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:09:21.731930 14616 tablet_bootstrap.cc:492] T c75a9f786c2b4d0382db67fa9cba274b P ed930b9cba294089a7183326762ac0f9: Bootstrap starting.
I20260620 00:09:21.732235 14612 raft_consensus.cc:348] T c75a9f786c2b4d0382db67fa9cba274b P ab60a5329cbf4d7995ad7e331de3529d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab60a5329cbf4d7995ad7e331de3529d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39455 } } peers { permanent_uuid: "ed930b9cba294089a7183326762ac0f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46639 } } peers { permanent_uuid: "5d26247cfb494b33bd2ecbe1494d3ea9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38605 } }
I20260620 00:09:21.732429 14612 raft_consensus.cc:374] T c75a9f786c2b4d0382db67fa9cba274b P ab60a5329cbf4d7995ad7e331de3529d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:09:21.732466 14612 raft_consensus.cc:729] T c75a9f786c2b4d0382db67fa9cba274b P ab60a5329cbf4d7995ad7e331de3529d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab60a5329cbf4d7995ad7e331de3529d, State: Initialized, Role: FOLLOWER
I20260620 00:09:21.732589 14616 tablet_bootstrap.cc:654] T c75a9f786c2b4d0382db67fa9cba274b P ed930b9cba294089a7183326762ac0f9: Neither blocks nor log segments found. Creating new log.
I20260620 00:09:21.732607 14612 consensus_queue.cc:260] T c75a9f786c2b4d0382db67fa9cba274b P ab60a5329cbf4d7995ad7e331de3529d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab60a5329cbf4d7995ad7e331de3529d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39455 } } peers { permanent_uuid: "ed930b9cba294089a7183326762ac0f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46639 } } peers { permanent_uuid: "5d26247cfb494b33bd2ecbe1494d3ea9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38605 } }
I20260620 00:09:21.732903 14612 ts_tablet_manager.cc:1428] T c75a9f786c2b4d0382db67fa9cba274b P ab60a5329cbf4d7995ad7e331de3529d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 00:09:21.732925 14327 heartbeater.cc:503] Master 127.0.98.62:39857 was elected leader, sending a full tablet report...
I20260620 00:09:21.733532 14616 log.cc:824] T c75a9f786c2b4d0382db67fa9cba274b P ed930b9cba294089a7183326762ac0f9: Log is configured to *not* fsync() on all Append() calls
I20260620 00:09:21.733845 14613 raft_consensus.cc:348] T c75a9f786c2b4d0382db67fa9cba274b P 5d26247cfb494b33bd2ecbe1494d3ea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab60a5329cbf4d7995ad7e331de3529d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39455 } } peers { permanent_uuid: "ed930b9cba294089a7183326762ac0f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46639 } } peers { permanent_uuid: "5d26247cfb494b33bd2ecbe1494d3ea9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38605 } }
I20260620 00:09:21.734019 14613 raft_consensus.cc:374] T c75a9f786c2b4d0382db67fa9cba274b P 5d26247cfb494b33bd2ecbe1494d3ea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:09:21.734066 14613 raft_consensus.cc:729] T c75a9f786c2b4d0382db67fa9cba274b P 5d26247cfb494b33bd2ecbe1494d3ea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d26247cfb494b33bd2ecbe1494d3ea9, State: Initialized, Role: FOLLOWER
I20260620 00:09:21.734196 14613 consensus_queue.cc:260] T c75a9f786c2b4d0382db67fa9cba274b P 5d26247cfb494b33bd2ecbe1494d3ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab60a5329cbf4d7995ad7e331de3529d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39455 } } peers { permanent_uuid: "ed930b9cba294089a7183326762ac0f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46639 } } peers { permanent_uuid: "5d26247cfb494b33bd2ecbe1494d3ea9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38605 } }
I20260620 00:09:21.734330 14616 tablet_bootstrap.cc:492] T c75a9f786c2b4d0382db67fa9cba274b P ed930b9cba294089a7183326762ac0f9: No bootstrap required, opened a new log
I20260620 00:09:21.734409 14616 ts_tablet_manager.cc:1397] T c75a9f786c2b4d0382db67fa9cba274b P ed930b9cba294089a7183326762ac0f9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 00:09:21.734504 14613 ts_tablet_manager.cc:1428] T c75a9f786c2b4d0382db67fa9cba274b P 5d26247cfb494b33bd2ecbe1494d3ea9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:09:21.734529 14461 heartbeater.cc:503] Master 127.0.98.62:39857 was elected leader, sending a full tablet report...
I20260620 00:09:21.736034 14616 raft_consensus.cc:348] T c75a9f786c2b4d0382db67fa9cba274b P ed930b9cba294089a7183326762ac0f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab60a5329cbf4d7995ad7e331de3529d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39455 } } peers { permanent_uuid: "ed930b9cba294089a7183326762ac0f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46639 } } peers { permanent_uuid: "5d26247cfb494b33bd2ecbe1494d3ea9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38605 } }
I20260620 00:09:21.736159 14616 raft_consensus.cc:374] T c75a9f786c2b4d0382db67fa9cba274b P ed930b9cba294089a7183326762ac0f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:09:21.736181 14616 raft_consensus.cc:729] T c75a9f786c2b4d0382db67fa9cba274b P ed930b9cba294089a7183326762ac0f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed930b9cba294089a7183326762ac0f9, State: Initialized, Role: FOLLOWER
I20260620 00:09:21.736285 14616 consensus_queue.cc:260] T c75a9f786c2b4d0382db67fa9cba274b P ed930b9cba294089a7183326762ac0f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab60a5329cbf4d7995ad7e331de3529d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39455 } } peers { permanent_uuid: "ed930b9cba294089a7183326762ac0f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46639 } } peers { permanent_uuid: "5d26247cfb494b33bd2ecbe1494d3ea9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38605 } }
I20260620 00:09:21.736568 14616 ts_tablet_manager.cc:1428] T c75a9f786c2b4d0382db67fa9cba274b P ed930b9cba294089a7183326762ac0f9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:09:21.736647 14595 heartbeater.cc:503] Master 127.0.98.62:39857 was elected leader, sending a full tablet report...
W20260620 00:09:21.769665 14462 tablet.cc:2369] T c75a9f786c2b4d0382db67fa9cba274b P 5d26247cfb494b33bd2ecbe1494d3ea9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:09:21.772078 14619 raft_consensus.cc:482] T c75a9f786c2b4d0382db67fa9cba274b P 5d26247cfb494b33bd2ecbe1494d3ea9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:09:21.772244 14619 raft_consensus.cc:504] T c75a9f786c2b4d0382db67fa9cba274b P 5d26247cfb494b33bd2ecbe1494d3ea9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab60a5329cbf4d7995ad7e331de3529d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39455 } } peers { permanent_uuid: "ed930b9cba294089a7183326762ac0f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46639 } } peers { permanent_uuid: "5d26247cfb494b33bd2ecbe1494d3ea9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38605 } }
I20260620 00:09:21.772662 14619 leader_election.cc:288] T c75a9f786c2b4d0382db67fa9cba274b P 5d26247cfb494b33bd2ecbe1494d3ea9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab60a5329cbf4d7995ad7e331de3529d (127.0.98.1:39455), ed930b9cba294089a7183326762ac0f9 (127.0.98.3:46639)
I20260620 00:09:21.775581 14463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:21.772877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46639 (local address 127.0.98.2:47937)
0620 00:09:21.772960 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:21.772974 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:21.773023 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:21.773177 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:21.773179 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:21.773181 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:09:21.773253 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:21.773255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:21.773988 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:21.773989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:21.774745 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:21.774747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:21.775304 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:21.775305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:21.775387 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:21.775390 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:09:21.775463 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 00:09:21.775507 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 00:09:21.775581 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:21.772986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47937 (local address 127.0.98.3:46639)
0620 00:09:21.773030 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:21.773032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:21.773044 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:09:21.773078 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:21.773078 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:21.773090 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:21.773162 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:09:21.773327 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:21.773878 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:21.774841 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:21.775223 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:21.775304 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:21.775314 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:21.775470 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 00:09:21.775507 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 00:09:21.775681 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:21.772978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57019 (local address 127.0.98.1:39455)
0620 00:09:21.773024 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:21.773026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:21.773184 (+ 158us) server_negotiation.cc:407] Connection header received
0620 00:09:21.773252 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:21.773253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:21.773299 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:21.773370 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:09:21.773544 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:21.774000 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:21.774938 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:21.775318 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:21.775402 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:21.775411 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:21.775571 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 00:09:21.775611 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 00:09:21.775899 14621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:21.772899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39455 (local address 127.0.98.2:57019)
0620 00:09:21.773089 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:21.773107 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:21.773190 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:21.773393 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:21.773394 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:21.773396 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:09:21.773471 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:21.773472 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:21.774084 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:21.774085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:21.774841 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:21.774843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:21.775403 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:21.775404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:21.775480 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:21.775484 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:09:21.775561 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 00:09:21.775602 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":127,"threads_started":1}
I20260620 00:09:21.775974 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75a9f786c2b4d0382db67fa9cba274b" candidate_uuid: "5d26247cfb494b33bd2ecbe1494d3ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed930b9cba294089a7183326762ac0f9" is_pre_election: true
I20260620 00:09:21.776161 14549 raft_consensus.cc:2454] T c75a9f786c2b4d0382db67fa9cba274b P ed930b9cba294089a7183326762ac0f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d26247cfb494b33bd2ecbe1494d3ea9 in term 0.
I20260620 00:09:21.776216 14281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75a9f786c2b4d0382db67fa9cba274b" candidate_uuid: "5d26247cfb494b33bd2ecbe1494d3ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab60a5329cbf4d7995ad7e331de3529d" is_pre_election: true
I20260620 00:09:21.776448 14350 leader_election.cc:302] T c75a9f786c2b4d0382db67fa9cba274b P 5d26247cfb494b33bd2ecbe1494d3ea9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d26247cfb494b33bd2ecbe1494d3ea9, ed930b9cba294089a7183326762ac0f9; no voters:
I20260620 00:09:21.776525 14281 raft_consensus.cc:2454] T c75a9f786c2b4d0382db67fa9cba274b P ab60a5329cbf4d7995ad7e331de3529d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d26247cfb494b33bd2ecbe1494d3ea9 in term 0.
I20260620 00:09:21.776578 14619 raft_consensus.cc:2781] T c75a9f786c2b4d0382db67fa9cba274b P 5d26247cfb494b33bd2ecbe1494d3ea9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:09:21.776635 14619 raft_consensus.cc:482] T c75a9f786c2b4d0382db67fa9cba274b P 5d26247cfb494b33bd2ecbe1494d3ea9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:09:21.776656 14619 raft_consensus.cc:3037] T c75a9f786c2b4d0382db67fa9cba274b P 5d26247cfb494b33bd2ecbe1494d3ea9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:09:21.777485 14619 raft_consensus.cc:504] T c75a9f786c2b4d0382db67fa9cba274b P 5d26247cfb494b33bd2ecbe1494d3ea9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab60a5329cbf4d7995ad7e331de3529d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39455 } } peers { permanent_uuid: "ed930b9cba294089a7183326762ac0f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46639 } } peers { permanent_uuid: "5d26247cfb494b33bd2ecbe1494d3ea9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38605 } }
I20260620 00:09:21.777628 14619 leader_election.cc:288] T c75a9f786c2b4d0382db67fa9cba274b P 5d26247cfb494b33bd2ecbe1494d3ea9 [CANDIDATE]: Term 1 election: Requested vote from peers ab60a5329cbf4d7995ad7e331de3529d (127.0.98.1:39455), ed930b9cba294089a7183326762ac0f9 (127.0.98.3:46639)
I20260620 00:09:21.777784 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75a9f786c2b4d0382db67fa9cba274b" candidate_uuid: "5d26247cfb494b33bd2ecbe1494d3ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed930b9cba294089a7183326762ac0f9"
I20260620 00:09:21.777823 14281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75a9f786c2b4d0382db67fa9cba274b" candidate_uuid: "5d26247cfb494b33bd2ecbe1494d3ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab60a5329cbf4d7995ad7e331de3529d"
I20260620 00:09:21.777889 14549 raft_consensus.cc:3037] T c75a9f786c2b4d0382db67fa9cba274b P ed930b9cba294089a7183326762ac0f9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:09:21.777887 14281 raft_consensus.cc:3037] T c75a9f786c2b4d0382db67fa9cba274b P ab60a5329cbf4d7995ad7e331de3529d [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:09:21.778609 14549 raft_consensus.cc:2454] T c75a9f786c2b4d0382db67fa9cba274b P ed930b9cba294089a7183326762ac0f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d26247cfb494b33bd2ecbe1494d3ea9 in term 1.
I20260620 00:09:21.778645 14281 raft_consensus.cc:2454] T c75a9f786c2b4d0382db67fa9cba274b P ab60a5329cbf4d7995ad7e331de3529d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d26247cfb494b33bd2ecbe1494d3ea9 in term 1.
I20260620 00:09:21.778823 14348 leader_election.cc:302] T c75a9f786c2b4d0382db67fa9cba274b P 5d26247cfb494b33bd2ecbe1494d3ea9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d26247cfb494b33bd2ecbe1494d3ea9, ab60a5329cbf4d7995ad7e331de3529d; no voters:
I20260620 00:09:21.778923 14619 raft_consensus.cc:2781] T c75a9f786c2b4d0382db67fa9cba274b P 5d26247cfb494b33bd2ecbe1494d3ea9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:09:21.779076 14619 raft_consensus.cc:686] T c75a9f786c2b4d0382db67fa9cba274b P 5d26247cfb494b33bd2ecbe1494d3ea9 [term 1 LEADER]: Becoming Leader. State: Replica: 5d26247cfb494b33bd2ecbe1494d3ea9, State: Running, Role: LEADER
I20260620 00:09:21.779182 14619 consensus_queue.cc:237] T c75a9f786c2b4d0382db67fa9cba274b P 5d26247cfb494b33bd2ecbe1494d3ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab60a5329cbf4d7995ad7e331de3529d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39455 } } peers { permanent_uuid: "ed930b9cba294089a7183326762ac0f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46639 } } peers { permanent_uuid: "5d26247cfb494b33bd2ecbe1494d3ea9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38605 } }
I20260620 00:09:21.780140 14136 catalog_manager.cc:5495] T c75a9f786c2b4d0382db67fa9cba274b P 5d26247cfb494b33bd2ecbe1494d3ea9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d26247cfb494b33bd2ecbe1494d3ea9 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "5d26247cfb494b33bd2ecbe1494d3ea9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab60a5329cbf4d7995ad7e331de3529d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed930b9cba294089a7183326762ac0f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d26247cfb494b33bd2ecbe1494d3ea9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38605 } health_report { overall_health: HEALTHY } } }
I20260620 00:09:21.800050 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:21.797329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59858 (local address 127.0.98.2:38605)
0620 00:09:21.797471 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:21.797474 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:21.797484 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:09:21.797530 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:21.797531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:21.797555 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:21.797621 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:09:21.797859 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:21.798324 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:21.799124 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:21.799323 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:21.799387 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:21.799667 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:09:21.799816 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:09:21.799894 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:21.799964 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 00:09:21.799984 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260620 00:09:21.802229 14281 raft_consensus.cc:1261] T c75a9f786c2b4d0382db67fa9cba274b P ab60a5329cbf4d7995ad7e331de3529d [term 1 FOLLOWER]: Refusing update from remote peer 5d26247cfb494b33bd2ecbe1494d3ea9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:09:21.802229 14549 raft_consensus.cc:1261] T c75a9f786c2b4d0382db67fa9cba274b P ed930b9cba294089a7183326762ac0f9 [term 1 FOLLOWER]: Refusing update from remote peer 5d26247cfb494b33bd2ecbe1494d3ea9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:09:21.802614 14622 consensus_queue.cc:1035] T c75a9f786c2b4d0382db67fa9cba274b P 5d26247cfb494b33bd2ecbe1494d3ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab60a5329cbf4d7995ad7e331de3529d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:09:21.802807 14619 consensus_queue.cc:1035] T c75a9f786c2b4d0382db67fa9cba274b P 5d26247cfb494b33bd2ecbe1494d3ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed930b9cba294089a7183326762ac0f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:09:21.807479 14136 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:09:21.807595 14136 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:09:21.808230 14626 mvcc.cc:205] Tried to move back new op lower bound from 7298720406738472960 to 7298720406648549376. Current Snapshot: MvccSnapshot[applied={T|T < 7298720406738472960 or (T in {7298720406738472960})}]
I20260620 00:09:21.808454 14136 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9: Sending DeleteTablet for 3 replicas of tablet c75a9f786c2b4d0382db67fa9cba274b
I20260620 00:09:21.808756 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 14198
I20260620 00:09:21.808771 14529 tablet_service.cc:1514] Processing DeleteTablet for tablet c75a9f786c2b4d0382db67fa9cba274b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:09:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38454
I20260620 00:09:21.808817 14395 tablet_service.cc:1514] Processing DeleteTablet for tablet c75a9f786c2b4d0382db67fa9cba274b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:09:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59856
I20260620 00:09:21.808974 14528 tablet_service.cc:1514] Processing DeleteTablet for tablet c75a9f786c2b4d0382db67fa9cba274b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:09:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38454
W20260620 00:09:21.813903 14120 connection.cc:535] client connection to 127.0.98.1:39455 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 00:09:21.814016 14120 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 00:09:21.814028 14635 tablet_replica.cc:330] T c75a9f786c2b4d0382db67fa9cba274b P 5d26247cfb494b33bd2ecbe1494d3ea9: stopping tablet replica
I20260620 00:09:21.814132 14635 raft_consensus.cc:2229] T c75a9f786c2b4d0382db67fa9cba274b P 5d26247cfb494b33bd2ecbe1494d3ea9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:09:21.814239 14635 raft_consensus.cc:2258] T c75a9f786c2b4d0382db67fa9cba274b P 5d26247cfb494b33bd2ecbe1494d3ea9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:09:21.814404 14634 tablet_replica.cc:330] T c75a9f786c2b4d0382db67fa9cba274b P ed930b9cba294089a7183326762ac0f9: stopping tablet replica
I20260620 00:09:21.814508 14634 raft_consensus.cc:2229] T c75a9f786c2b4d0382db67fa9cba274b P ed930b9cba294089a7183326762ac0f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:09:21.814626 14634 raft_consensus.cc:2258] T c75a9f786c2b4d0382db67fa9cba274b P ed930b9cba294089a7183326762ac0f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:09:21.814779 14635 ts_tablet_manager.cc:1905] T c75a9f786c2b4d0382db67fa9cba274b P 5d26247cfb494b33bd2ecbe1494d3ea9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 00:09:21.814883 14606 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:09:21.814516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39455 (local address 127.0.0.1:36958)
0620 00:09:21.814570 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:21.814583 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:21.814625 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:21.814839 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39455: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 00:09:21.815106 14120 catalog_manager.cc:4584] TS ab60a5329cbf4d7995ad7e331de3529d (127.0.98.1:39455): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c75a9f786c2b4d0382db67fa9cba274b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39455: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 00:09:21.815140 14634 ts_tablet_manager.cc:1905] T c75a9f786c2b4d0382db67fa9cba274b P ed930b9cba294089a7183326762ac0f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:09:21.815621 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 14332
I20260620 00:09:21.817193 14634 ts_tablet_manager.cc:1918] T c75a9f786c2b4d0382db67fa9cba274b P ed930b9cba294089a7183326762ac0f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:09:21.817281 14634 log.cc:1196] T c75a9f786c2b4d0382db67fa9cba274b P ed930b9cba294089a7183326762ac0f9: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/wal/wals/c75a9f786c2b4d0382db67fa9cba274b
I20260620 00:09:21.817600 14634 ts_tablet_manager.cc:1939] T c75a9f786c2b4d0382db67fa9cba274b P ed930b9cba294089a7183326762ac0f9: Deleting consensus metadata
I20260620 00:09:21.818099 14122 catalog_manager.cc:4841] TS ed930b9cba294089a7183326762ac0f9 (127.0.98.3:46639): tablet c75a9f786c2b4d0382db67fa9cba274b (table test-table [id=bbefd1009fd849c2aa4b1b124b0ba71f]) successfully deleted
W20260620 00:09:21.818271 14122 catalog_manager.cc:4808] TS ed930b9cba294089a7183326762ac0f9 (127.0.98.3:46639): delete failed for tablet c75a9f786c2b4d0382db67fa9cba274b because the tablet was not found. No further retry: Not found: Tablet not found: c75a9f786c2b4d0382db67fa9cba274b
I20260620 00:09:21.821250 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 14466
I20260620 00:09:21.821915 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:21.821307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38605 (local address 127.0.0.1:59868)
0620 00:09:21.821826 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:21.821871 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499}
I20260620 00:09:21.826478 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 14104
I20260620 00:09:21.831736 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39857
--webserver_interface=127.0.98.62
--webserver_port=41495
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39857
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:09:21.913504 14637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:09:21.913770 14637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:09:21.913810 14637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:09:21.915010 14637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:09:21.915086 14637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:09:21.915109 14637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:09:21.915130 14637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:09:21.915159 14637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:09:21.916715 14637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:09:21.917006 14637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:09:21.917253 14637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:09:21.927953 14644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:21.928068 14643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:21.927953 14646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:09:21.929577 14637 server_base.cc:1034] running on GCE node
I20260620 00:09:21.929895 14637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:09:21.930318 14637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:09:21.931479 14637 hybrid_clock.cc:648] HybridClock initialized: now 1781914161931470 us; error 29 us; skew 500 ppm
Jun 20 00:09:21 dist-test-slave-grqt krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:09:21.934092 14637 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:09:21.934563 14637 webserver.cc:460] Webserver started at http://127.0.98.62:41495/ using document root <none> and password file <none>
I20260620 00:09:21.934803 14637 fs_manager.cc:359] Metadata directory not provided
I20260620 00:09:21.934883 14637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:09:21.936311 14637 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:09:21.937055 14653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:21.937271 14637 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:09:21.937393 14637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "97eabdebe29c4620b9981c6b4e3592e9"
format_stamp: "Formatted at 2026-06-20 00:09:21 on dist-test-slave-grqt"
server_key: "00e0c4144e4770b9adba18c0a6558286"
server_key_iv: "64768d8a24eb34a7d8cd44a62787f941"
server_key_version: "encryptionkey@0"
I20260620 00:09:21.937816 14637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:09:21.958081 14637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:09:21.960187 14637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:09:21.960366 14637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:09:21.964910 14637 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39857
I20260620 00:09:21.964926 14705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39857 every 8 connection(s)
I20260620 00:09:21.965349 14637 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:09:21.966974 14706 sys_catalog.cc:263] Verifying existing consensus state
I20260620 00:09:21.967401 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 14637
I20260620 00:09:21.967685 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39455
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39857
--webserver_port=45377
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:09:21.967712 14706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9: Bootstrap starting.
I20260620 00:09:21.978329 14706 log.cc:824] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9: Log is configured to *not* fsync() on all Append() calls
I20260620 00:09:21.980907 14706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 00:09:21.981249 14706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9: Bootstrap complete.
I20260620 00:09:21.984527 14706 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97eabdebe29c4620b9981c6b4e3592e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39857 } }
I20260620 00:09:21.984841 14706 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97eabdebe29c4620b9981c6b4e3592e9, State: Initialized, Role: FOLLOWER
I20260620 00:09:21.985009 14706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97eabdebe29c4620b9981c6b4e3592e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39857 } }
I20260620 00:09:21.985072 14706 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:09:21.985121 14706 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:09:21.985165 14706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 00:09:21.986517 14706 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97eabdebe29c4620b9981c6b4e3592e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39857 } }
I20260620 00:09:21.986624 14706 leader_election.cc:302] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97eabdebe29c4620b9981c6b4e3592e9; no voters:
I20260620 00:09:21.986785 14706 leader_election.cc:288] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 00:09:21.986856 14710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 00:09:21.987030 14710 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [term 2 LEADER]: Becoming Leader. State: Replica: 97eabdebe29c4620b9981c6b4e3592e9, State: Running, Role: LEADER
I20260620 00:09:21.987030 14706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:09:21.987322 14710 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97eabdebe29c4620b9981c6b4e3592e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39857 } }
I20260620 00:09:21.987648 14712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97eabdebe29c4620b9981c6b4e3592e9. Latest consensus state: current_term: 2 leader_uuid: "97eabdebe29c4620b9981c6b4e3592e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97eabdebe29c4620b9981c6b4e3592e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39857 } } }
I20260620 00:09:21.987650 14711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "97eabdebe29c4620b9981c6b4e3592e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97eabdebe29c4620b9981c6b4e3592e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39857 } } }
I20260620 00:09:21.987756 14712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [sys.catalog]: This master's current role is: LEADER
I20260620 00:09:21.987784 14711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [sys.catalog]: This master's current role is: LEADER
I20260620 00:09:21.988030 14719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:09:21.988875 14719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:09:21.988977 14719 catalog_manager.cc:1259] Loaded cluster ID: 4b1958d88f7b469eb7dd03fa0979b716
I20260620 00:09:21.989008 14719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:09:21.989394 14719 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 00:09:21.994055 14719 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 00:09:21.996615 14719 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:09:21.996845 14719 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9: Loaded TSK: 0
I20260620 00:09:21.997067 14719 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 00:09:22.057902 14708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:09:22.058156 14708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:09:22.058197 14708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:09:22.059477 14708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:09:22.059554 14708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:09:22.059602 14708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:09:22.061110 14708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39455
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45377
--tserver_master_addrs=127.0.98.62:39857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:09:22.061461 14708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:09:22.061712 14708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:09:22.067101 14733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:22.067170 14736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:09:22.067437 14708 server_base.cc:1034] running on GCE node
W20260620 00:09:22.067159 14734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:09:22.067726 14708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:09:22.067976 14708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:09:22.069154 14708 hybrid_clock.cc:648] HybridClock initialized: now 1781914162069140 us; error 37 us; skew 500 ppm
Jun 20 00:09:22 dist-test-slave-grqt krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:09:22.072554 14708 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:09:22.073112 14708 webserver.cc:460] Webserver started at http://127.0.98.1:45377/ using document root <none> and password file <none>
I20260620 00:09:22.073426 14708 fs_manager.cc:359] Metadata directory not provided
I20260620 00:09:22.073513 14708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:09:22.075309 14708 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 00:09:22.076079 14743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:22.076256 14708 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 00:09:22.076362 14708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/wal
uuid: "ab60a5329cbf4d7995ad7e331de3529d"
format_stamp: "Formatted at 2026-06-20 00:09:21 on dist-test-slave-grqt"
server_key: "18d71d70ac1afa913059b328807c0be7"
server_key_iv: "bc6bdfa337a99afc268ef29bfdb7eaf2"
server_key_version: "encryptionkey@0"
I20260620 00:09:22.076815 14708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:09:22.101526 14708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:09:22.104298 14708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:09:22.104470 14708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:09:22.104749 14708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:09:22.105245 14750 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 00:09:22.106777 14708 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 00:09:22.106844 14708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 00:09:22.106881 14708 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 00:09:22.107755 14708 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 00:09:22.107802 14708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 00:09:22.107913 14750 tablet_bootstrap.cc:492] T c75a9f786c2b4d0382db67fa9cba274b P ab60a5329cbf4d7995ad7e331de3529d: Bootstrap starting.
I20260620 00:09:22.114795 14708 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39455
I20260620 00:09:22.115075 14857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39455 every 8 connection(s)
I20260620 00:09:22.115231 14708 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:09:22.119108 14750 log.cc:824] T c75a9f786c2b4d0382db67fa9cba274b P ab60a5329cbf4d7995ad7e331de3529d: Log is configured to *not* fsync() on all Append() calls
I20260620 00:09:22.121361 14750 tablet_bootstrap.cc:492] T c75a9f786c2b4d0382db67fa9cba274b P ab60a5329cbf4d7995ad7e331de3529d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 00:09:22.121686 14750 tablet_bootstrap.cc:492] T c75a9f786c2b4d0382db67fa9cba274b P ab60a5329cbf4d7995ad7e331de3529d: Bootstrap complete.
I20260620 00:09:22.122845 14750 ts_tablet_manager.cc:1397] T c75a9f786c2b4d0382db67fa9cba274b P ab60a5329cbf4d7995ad7e331de3529d: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20260620 00:09:22.123694 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 14708
I20260620 00:09:22.123982 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38605
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39857
--webserver_port=36799
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 00:09:22 dist-test-slave-grqt krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:09:22.124172 14750 raft_consensus.cc:348] T c75a9f786c2b4d0382db67fa9cba274b P ab60a5329cbf4d7995ad7e331de3529d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab60a5329cbf4d7995ad7e331de3529d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39455 } } peers { permanent_uuid: "ed930b9cba294089a7183326762ac0f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46639 } } peers { permanent_uuid: "5d26247cfb494b33bd2ecbe1494d3ea9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38605 } }
I20260620 00:09:22.124547 14750 raft_consensus.cc:729] T c75a9f786c2b4d0382db67fa9cba274b P ab60a5329cbf4d7995ad7e331de3529d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab60a5329cbf4d7995ad7e331de3529d, State: Initialized, Role: FOLLOWER
I20260620 00:09:22.124707 14750 consensus_queue.cc:260] T c75a9f786c2b4d0382db67fa9cba274b P ab60a5329cbf4d7995ad7e331de3529d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab60a5329cbf4d7995ad7e331de3529d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39455 } } peers { permanent_uuid: "ed930b9cba294089a7183326762ac0f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46639 } } peers { permanent_uuid: "5d26247cfb494b33bd2ecbe1494d3ea9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38605 } }
I20260620 00:09:22.125000 14750 ts_tablet_manager.cc:1428] T c75a9f786c2b4d0382db67fa9cba274b P ab60a5329cbf4d7995ad7e331de3529d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 00:09:22.128851 14860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:22.115975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.1:35901)
0620 00:09:22.116177 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:22.116224 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:22.116836 (+ 612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:22.117442 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:22.117446 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:22.117819 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:09:22.118149 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:22.118156 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:22.119831 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:22.119832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:22.121126 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:22.121130 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:22.122025 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:22.122026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:22.122172 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:22.122852 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:09:22.122863 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:09:22.124765 (+ 1902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:09:22.126978 (+ 2213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:22.126985 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:22.126987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:22.127219 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:22.127497 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:22.127498 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:22.127499 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:22.127606 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:22.128083 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:09:22.128090 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:09:22.128311 (+ 221us) client_negotiation.cc:769] Sending connection context
0620 00:09:22.128449 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 00:09:22.128636 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":107,"threads_started":1}
I20260620 00:09:22.129457 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:22.116072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35901 (local address 127.0.98.62:39857)
0620 00:09:22.116315 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:22.116329 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:22.116377 (+ 48us) server_negotiation.cc:407] Connection header received
0620 00:09:22.117093 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:22.117101 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:22.117186 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:22.117393 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:09:22.118310 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:22.119710 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:22.121249 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:22.121907 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:22.122018 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:22.124909 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:09:22.124915 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:09:22.124915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:09:22.124931 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:09:22.126845 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:22.127319 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:22.127322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:22.127323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:22.127422 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:22.127678 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:22.127679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:22.127680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:22.127988 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:09:22.128101 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:22.128540 (+ 439us) server_negotiation.cc:299] Negotiation successful
0620 00:09:22.128722 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":77,"threads_started":1}
I20260620 00:09:22.130242 14858 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39857
I20260620 00:09:22.130376 14858 heartbeater.cc:464] Registering TS with master...
I20260620 00:09:22.130561 14858 heartbeater.cc:511] Master 127.0.98.62:39857 requested a full tablet report, sending...
I20260620 00:09:22.131304 14670 ts_manager.cc:194] Registered new tserver with Master: ab60a5329cbf4d7995ad7e331de3529d (127.0.98.1:39455)
I20260620 00:09:22.132426 14670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35901
I20260620 00:09:22.133412 14858 heartbeater.cc:503] Master 127.0.98.62:39857 was elected leader, sending a full tablet report...
Jun 20 00:09:22 dist-test-slave-grqt krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 00:09:22.140872 14866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:22.132703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36968 (local address 127.0.98.1:39455)
0620 00:09:22.132900 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:22.132905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:22.132916 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:09:22.132956 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:22.132958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:22.132978 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:22.133060 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:09:22.133687 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:22.134413 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:22.135292 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:22.135597 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:22.135678 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:22.137727 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:09:22.137736 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:09:22.137737 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:09:22.137748 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:09:22.139439 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:22.139935 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:22.139938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:22.139939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:22.140005 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:22.140225 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:22.140226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:22.140226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:22.140422 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:09:22.140500 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:22.140677 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 00:09:22.140762 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":160,"thread_start_us":112,"threads_started":1}
I20260620 00:09:22.140878 14865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:22.131921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39455 (local address 127.0.0.1:36968)
0620 00:09:22.132090 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:22.132126 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:22.132191 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:22.133074 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:22.133080 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:22.133448 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:09:22.133601 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:22.133604 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:22.134505 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:22.134506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:22.135199 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:22.135202 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:22.135675 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:22.135679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:22.135780 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:22.136288 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:09:22.136292 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:09:22.137618 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:09:22.139548 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:22.139556 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:22.139558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:22.139758 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:22.140081 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:22.140082 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:22.140082 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:22.140150 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:22.140478 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:09:22.140484 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:09:22.140535 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 00:09:22.140641 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 00:09:22.140764 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":91,"threads_started":1}
I20260620 00:09:22.141680 14792 tablet_service.cc:1514] Processing DeleteTablet for tablet c75a9f786c2b4d0382db67fa9cba274b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:09:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36968
I20260620 00:09:22.142208 14867 tablet_replica.cc:330] stopping tablet replica
I20260620 00:09:22.142318 14867 raft_consensus.cc:2229] T c75a9f786c2b4d0382db67fa9cba274b P ab60a5329cbf4d7995ad7e331de3529d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:09:22.142386 14867 raft_consensus.cc:2258] T c75a9f786c2b4d0382db67fa9cba274b P ab60a5329cbf4d7995ad7e331de3529d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:09:22.142715 14867 ts_tablet_manager.cc:1905] T c75a9f786c2b4d0382db67fa9cba274b P ab60a5329cbf4d7995ad7e331de3529d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:09:22.144891 14867 ts_tablet_manager.cc:1918] T c75a9f786c2b4d0382db67fa9cba274b P ab60a5329cbf4d7995ad7e331de3529d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:09:22.144984 14867 log.cc:1196] T c75a9f786c2b4d0382db67fa9cba274b P ab60a5329cbf4d7995ad7e331de3529d: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/wal/wals/c75a9f786c2b4d0382db67fa9cba274b
I20260620 00:09:22.145210 14867 ts_tablet_manager.cc:1939] T c75a9f786c2b4d0382db67fa9cba274b P ab60a5329cbf4d7995ad7e331de3529d: Deleting consensus metadata
I20260620 00:09:22.145750 14654 catalog_manager.cc:4841] TS ab60a5329cbf4d7995ad7e331de3529d (127.0.98.1:39455): tablet c75a9f786c2b4d0382db67fa9cba274b (table test-table [id=bbefd1009fd849c2aa4b1b124b0ba71f]) successfully deleted
W20260620 00:09:22.215696 14864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:09:22.215955 14864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:09:22.215996 14864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:09:22.217260 14864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:09:22.217334 14864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:09:22.217377 14864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:09:22.218884 14864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38605
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36799
--tserver_master_addrs=127.0.98.62:39857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:09:22.219161 14864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:09:22.219398 14864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:09:22.224550 14875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:22.224681 14872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:22.224555 14873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:09:22.224597 14864 server_base.cc:1034] running on GCE node
I20260620 00:09:22.225018 14864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:09:22.225282 14864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:09:22.226545 14864 hybrid_clock.cc:648] HybridClock initialized: now 1781914162226531 us; error 35 us; skew 500 ppm
Jun 20 00:09:22 dist-test-slave-grqt krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:09:22.229120 14864 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:09:22.229694 14864 webserver.cc:460] Webserver started at http://127.0.98.2:36799/ using document root <none> and password file <none>
I20260620 00:09:22.229947 14864 fs_manager.cc:359] Metadata directory not provided
I20260620 00:09:22.230031 14864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:09:22.231444 14864 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:09:22.232201 14882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:22.232400 14864 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:09:22.232508 14864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/wal
uuid: "5d26247cfb494b33bd2ecbe1494d3ea9"
format_stamp: "Formatted at 2026-06-20 00:09:21 on dist-test-slave-grqt"
server_key: "57f2e713c9b24fbb66ef98a1b8cb90af"
server_key_iv: "1db291533152684dc6d9581b19a58b58"
server_key_version: "encryptionkey@0"
I20260620 00:09:22.232872 14864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:09:22.242552 14864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:09:22.244386 14864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:09:22.244545 14864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:09:22.244776 14864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:09:22.245180 14889 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260620 00:09:22.246580 14889 ts_tablet_manager.cc:1852] T c75a9f786c2b4d0382db67fa9cba274b P 5d26247cfb494b33bd2ecbe1494d3ea9: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260620 00:09:22.246843 14889 ts_tablet_manager.cc:1905] T c75a9f786c2b4d0382db67fa9cba274b P 5d26247cfb494b33bd2ecbe1494d3ea9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:09:22.249017 14889 ts_tablet_manager.cc:1918] T c75a9f786c2b4d0382db67fa9cba274b P 5d26247cfb494b33bd2ecbe1494d3ea9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:09:22.249085 14889 log.cc:1196] T c75a9f786c2b4d0382db67fa9cba274b P 5d26247cfb494b33bd2ecbe1494d3ea9: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/wal/wals/c75a9f786c2b4d0382db67fa9cba274b
I20260620 00:09:22.249341 14889 ts_tablet_manager.cc:1939] T c75a9f786c2b4d0382db67fa9cba274b P 5d26247cfb494b33bd2ecbe1494d3ea9: Deleting consensus metadata
I20260620 00:09:22.249603 14864 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260620 00:09:22.249662 14864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260620 00:09:22.249727 14864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:09:22.249758 14864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:22.255932 14864 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38605
I20260620 00:09:22.255977 14996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38605 every 8 connection(s)
I20260620 00:09:22.256330 14864 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:09:22.257050 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 14864
I20260620 00:09:22.257354 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46639
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39857
--webserver_port=39401
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 00:09:22 dist-test-slave-grqt krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:09:22.267642 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:22.257075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57957 (local address 127.0.98.62:39857)
0620 00:09:22.257148 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:22.257150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:22.257442 (+ 292us) server_negotiation.cc:407] Connection header received
0620 00:09:22.258251 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:22.258253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:22.258267 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:22.258326 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:09:22.259110 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:22.259920 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:22.261545 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:22.261744 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:22.261811 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:22.264617 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:09:22.264618 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:09:22.264619 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:09:22.264628 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:09:22.266172 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:22.266637 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:22.266639 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:22.266639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:22.266700 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:22.266966 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:22.266966 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:22.266967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:22.267080 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:09:22.267154 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:22.267457 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 00:09:22.267545 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 00:09:22.267827 14999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:22.257063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.2:57957)
0620 00:09:22.257287 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:22.257333 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:22.258144 (+ 811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:22.258397 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:22.258401 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:22.258793 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:09:22.259016 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:22.259021 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:22.260016 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:22.260018 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:22.261460 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:22.261463 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:22.261840 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:22.261841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:22.262018 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:22.262594 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:09:22.262606 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:09:22.264512 (+ 1906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:09:22.266302 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:22.266308 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:22.266311 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:22.266544 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:22.266785 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:22.266785 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:22.266786 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:22.266898 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:22.267157 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:09:22.267164 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:09:22.267316 (+ 152us) client_negotiation.cc:769] Sending connection context
0620 00:09:22.267426 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 00:09:22.267587 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":108,"threads_started":1}
I20260620 00:09:22.268352 14997 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39857
I20260620 00:09:22.268458 14997 heartbeater.cc:464] Registering TS with master...
I20260620 00:09:22.268641 14997 heartbeater.cc:511] Master 127.0.98.62:39857 requested a full tablet report, sending...
I20260620 00:09:22.269062 14670 ts_manager.cc:194] Registered new tserver with Master: 5d26247cfb494b33bd2ecbe1494d3ea9 (127.0.98.2:38605)
I20260620 00:09:22.269585 14670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57957
W20260620 00:09:22.341665 15000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:09:22.341879 15000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:09:22.341908 15000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:09:22.343061 15000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:09:22.343101 15000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:09:22.343127 15000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:09:22.344551 15000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46639
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39401
--tserver_master_addrs=127.0.98.62:39857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:09:22.344753 15000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:09:22.344939 15000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:09:22.350302 15006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:09:22.350461 15000 server_base.cc:1034] running on GCE node
W20260620 00:09:22.350531 15005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:22.350359 15008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:09:22.350811 15000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:09:22.351073 15000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:09:22.352222 15000 hybrid_clock.cc:648] HybridClock initialized: now 1781914162352209 us; error 30 us; skew 500 ppm
Jun 20 00:09:22 dist-test-slave-grqt krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:09:22.354749 15000 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:09:22.355182 15000 webserver.cc:460] Webserver started at http://127.0.98.3:39401/ using document root <none> and password file <none>
I20260620 00:09:22.355444 15000 fs_manager.cc:359] Metadata directory not provided
I20260620 00:09:22.355533 15000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:09:22.356901 15000 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:09:22.357609 15015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:22.357833 15000 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:09:22.357936 15000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/wal
uuid: "ed930b9cba294089a7183326762ac0f9"
format_stamp: "Formatted at 2026-06-20 00:09:21 on dist-test-slave-grqt"
server_key: "f3c2ede53150f0847211bae243c9f748"
server_key_iv: "2aab5a610bd19d23eb3f821cc107fb3f"
server_key_version: "encryptionkey@0"
I20260620 00:09:22.358309 15000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:09:22.368278 15000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:09:22.370405 15000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:09:22.370569 15000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:09:22.370805 15000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:09:22.371119 15000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:09:22.371174 15000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:22.371217 15000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:09:22.371243 15000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:22.377239 15000 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46639
I20260620 00:09:22.377281 15128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46639 every 8 connection(s)
I20260620 00:09:22.377672 15000 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:09:22.383124 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 15000
Jun 20 00:09:22 dist-test-slave-grqt krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:09:22.389513 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:22.378412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39125 (local address 127.0.98.62:39857)
0620 00:09:22.378500 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:22.378503 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:22.378788 (+ 285us) server_negotiation.cc:407] Connection header received
0620 00:09:22.379247 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:22.379248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:22.379266 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:22.379330 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:09:22.380250 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:22.380840 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:22.381969 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:22.382166 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:22.382227 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:22.384946 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:09:22.384948 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:09:22.384948 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:09:22.384956 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:09:22.386644 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:22.388164 (+ 1520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:22.388170 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:22.388170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:22.388226 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:22.388555 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:22.388556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:22.388556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:22.388707 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:09:22.388802 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:22.389218 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 00:09:22.389353 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 00:09:22.389777 15131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:22.378396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.3:39125)
0620 00:09:22.378592 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:22.378630 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:22.379151 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:22.379385 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:22.379389 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:22.379956 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:09:22.380151 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:22.380155 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:22.380922 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:22.380923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:22.381894 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:22.381897 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:22.382295 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:22.382296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:22.382412 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:22.382964 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:09:22.382976 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:09:22.384844 (+ 1868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:09:22.387676 (+ 2832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:22.387686 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:22.387689 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:22.388076 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:22.388305 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:22.388306 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:22.388306 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:22.388482 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:22.388784 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:09:22.388790 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:09:22.388997 (+ 207us) client_negotiation.cc:769] Sending connection context
0620 00:09:22.389193 (+ 196us) client_negotiation.cc:240] Negotiation successful
0620 00:09:22.389465 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":110,"threads_started":1}
I20260620 00:09:22.390297 15129 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39857
I20260620 00:09:22.390409 15129 heartbeater.cc:464] Registering TS with master...
I20260620 00:09:22.390623 15129 heartbeater.cc:511] Master 127.0.98.62:39857 requested a full tablet report, sending...
I20260620 00:09:22.390874 15133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:22.383587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35612 (local address 127.0.98.62:39857)
0620 00:09:22.383713 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:22.383714 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:22.383722 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:09:22.383794 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:22.383794 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:22.383805 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:22.383866 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:09:22.384607 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:22.385096 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:22.385926 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:22.386125 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:22.386201 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:22.387649 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:09:22.387651 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:09:22.387651 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:09:22.387661 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:09:22.389567 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:22.390010 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:22.390011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:22.390011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:22.390075 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:22.390320 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:22.390321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:22.390321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:22.390475 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:09:22.390642 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:22.390686 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 00:09:22.390771 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":58,"threads_started":1}
I20260620 00:09:22.391100 14670 ts_manager.cc:194] Registered new tserver with Master: ed930b9cba294089a7183326762ac0f9 (127.0.98.3:46639)
I20260620 00:09:22.391481 14670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39125
I20260620 00:09:22.391644 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:09:22.394776 15133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:22.392008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35622 (local address 127.0.98.62:39857)
0620 00:09:22.392048 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:22.392050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:22.392166 (+ 116us) server_negotiation.cc:407] Connection header received
0620 00:09:22.392203 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:22.392204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:22.392242 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:22.392340 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:09:22.392419 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:22.392918 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:22.393958 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:22.394158 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:22.394219 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:22.394408 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:09:22.394504 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:09:22.394570 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:22.394705 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 00:09:22.394724 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 00:09:22.395071 14670 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:35622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:09:22.396698 14670 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:09:22.399305 14792 tablet_service.cc:1467] Processing CreateTablet for tablet 5f3c551e427f46ba9a5c01bc97f7fa07 (DEFAULT_TABLE table=test-table [id=d5a043ce560344a095599c5db9583d81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:09:22.399536 14792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f3c551e427f46ba9a5c01bc97f7fa07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:09:22.401052 14750 tablet_bootstrap.cc:492] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d: Bootstrap starting.
I20260620 00:09:22.401432 14750 tablet_bootstrap.cc:654] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d: Neither blocks nor log segments found. Creating new log.
I20260620 00:09:22.402014 14750 tablet_bootstrap.cc:492] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d: No bootstrap required, opened a new log
I20260620 00:09:22.402073 14750 ts_tablet_manager.cc:1397] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 00:09:22.402424 14750 raft_consensus.cc:348] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed930b9cba294089a7183326762ac0f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46639 } } peers { permanent_uuid: "5d26247cfb494b33bd2ecbe1494d3ea9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38605 } } peers { permanent_uuid: "ab60a5329cbf4d7995ad7e331de3529d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39455 } }
I20260620 00:09:22.402514 14750 raft_consensus.cc:374] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:09:22.402573 14750 raft_consensus.cc:729] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab60a5329cbf4d7995ad7e331de3529d, State: Initialized, Role: FOLLOWER
I20260620 00:09:22.402631 14750 consensus_queue.cc:260] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed930b9cba294089a7183326762ac0f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46639 } } peers { permanent_uuid: "5d26247cfb494b33bd2ecbe1494d3ea9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38605 } } peers { permanent_uuid: "ab60a5329cbf4d7995ad7e331de3529d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39455 } }
I20260620 00:09:22.402725 14750 ts_tablet_manager.cc:1428] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 00:09:22.402734 14865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:22.398533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46639 (local address 127.0.0.1:38458)
0620 00:09:22.398591 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:22.398604 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:22.398644 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:22.399007 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:22.399008 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:22.399009 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:09:22.399059 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:22.399061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:22.400419 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:22.400419 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:22.401644 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:22.401646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:22.402428 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:22.402429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:22.402510 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:22.402532 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 00:09:22.402588 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 00:09:22.402644 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":44}
I20260620 00:09:22.403076 15136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:22.398599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38458 (local address 127.0.98.3:46639)
0620 00:09:22.398803 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:22.398810 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:22.398821 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:09:22.398875 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:22.398878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:22.398895 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:22.399005 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:09:22.399122 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:22.400272 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:22.401714 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:22.402323 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:22.402430 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:22.402493 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:22.402924 (+ 431us) server_negotiation.cc:299] Negotiation successful
0620 00:09:22.402976 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":88,"threads_started":1}
I20260620 00:09:22.403605 15137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:22.398900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59882 (local address 127.0.98.2:38605)
0620 00:09:22.399068 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:22.399072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:22.399081 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:09:22.399120 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:22.399121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:22.399138 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:22.399231 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:09:22.399345 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:22.400267 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:22.402173 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:22.403046 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:22.403121 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:22.403413 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:22.403511 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 00:09:22.403544 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":108,"threads_started":1}
I20260620 00:09:22.403620 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:22.398575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38605 (local address 127.0.0.1:59882)
0620 00:09:22.398756 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:22.398767 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:22.398815 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:22.399230 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:22.399231 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:22.399233 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:09:22.399280 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:22.399282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:22.400376 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:22.400377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:22.401992 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:22.401995 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:22.403140 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:22.403141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:22.403425 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:22.403431 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 00:09:22.403503 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 00:09:22.403547 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":31,"thread_start_us":101,"threads_started":1}
I20260620 00:09:22.403921 15063 tablet_service.cc:1467] Processing CreateTablet for tablet 5f3c551e427f46ba9a5c01bc97f7fa07 (DEFAULT_TABLE table=test-table [id=d5a043ce560344a095599c5db9583d81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:09:22.404173 14931 tablet_service.cc:1467] Processing CreateTablet for tablet 5f3c551e427f46ba9a5c01bc97f7fa07 (DEFAULT_TABLE table=test-table [id=d5a043ce560344a095599c5db9583d81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:09:22.404369 15063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f3c551e427f46ba9a5c01bc97f7fa07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:09:22.404531 14931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f3c551e427f46ba9a5c01bc97f7fa07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:09:22.406919 14889 tablet_bootstrap.cc:492] T 5f3c551e427f46ba9a5c01bc97f7fa07 P 5d26247cfb494b33bd2ecbe1494d3ea9: Bootstrap starting.
I20260620 00:09:22.407773 14889 tablet_bootstrap.cc:654] T 5f3c551e427f46ba9a5c01bc97f7fa07 P 5d26247cfb494b33bd2ecbe1494d3ea9: Neither blocks nor log segments found. Creating new log.
I20260620 00:09:22.408097 14889 log.cc:824] T 5f3c551e427f46ba9a5c01bc97f7fa07 P 5d26247cfb494b33bd2ecbe1494d3ea9: Log is configured to *not* fsync() on all Append() calls
I20260620 00:09:22.408579 15139 tablet_bootstrap.cc:492] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ed930b9cba294089a7183326762ac0f9: Bootstrap starting.
I20260620 00:09:22.408902 14889 tablet_bootstrap.cc:492] T 5f3c551e427f46ba9a5c01bc97f7fa07 P 5d26247cfb494b33bd2ecbe1494d3ea9: No bootstrap required, opened a new log
I20260620 00:09:22.408973 14889 ts_tablet_manager.cc:1397] T 5f3c551e427f46ba9a5c01bc97f7fa07 P 5d26247cfb494b33bd2ecbe1494d3ea9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260620 00:09:22.409314 15139 tablet_bootstrap.cc:654] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ed930b9cba294089a7183326762ac0f9: Neither blocks nor log segments found. Creating new log.
I20260620 00:09:22.409989 15139 log.cc:824] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ed930b9cba294089a7183326762ac0f9: Log is configured to *not* fsync() on all Append() calls
I20260620 00:09:22.410871 15139 tablet_bootstrap.cc:492] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ed930b9cba294089a7183326762ac0f9: No bootstrap required, opened a new log
I20260620 00:09:22.410985 15139 ts_tablet_manager.cc:1397] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ed930b9cba294089a7183326762ac0f9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:09:22.411334 14889 raft_consensus.cc:348] T 5f3c551e427f46ba9a5c01bc97f7fa07 P 5d26247cfb494b33bd2ecbe1494d3ea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed930b9cba294089a7183326762ac0f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46639 } } peers { permanent_uuid: "5d26247cfb494b33bd2ecbe1494d3ea9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38605 } } peers { permanent_uuid: "ab60a5329cbf4d7995ad7e331de3529d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39455 } }
I20260620 00:09:22.411502 14889 raft_consensus.cc:374] T 5f3c551e427f46ba9a5c01bc97f7fa07 P 5d26247cfb494b33bd2ecbe1494d3ea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:09:22.411552 14889 raft_consensus.cc:729] T 5f3c551e427f46ba9a5c01bc97f7fa07 P 5d26247cfb494b33bd2ecbe1494d3ea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d26247cfb494b33bd2ecbe1494d3ea9, State: Initialized, Role: FOLLOWER
I20260620 00:09:22.411684 14889 consensus_queue.cc:260] T 5f3c551e427f46ba9a5c01bc97f7fa07 P 5d26247cfb494b33bd2ecbe1494d3ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed930b9cba294089a7183326762ac0f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46639 } } peers { permanent_uuid: "5d26247cfb494b33bd2ecbe1494d3ea9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38605 } } peers { permanent_uuid: "ab60a5329cbf4d7995ad7e331de3529d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39455 } }
I20260620 00:09:22.412034 14889 ts_tablet_manager.cc:1428] T 5f3c551e427f46ba9a5c01bc97f7fa07 P 5d26247cfb494b33bd2ecbe1494d3ea9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:09:22.412046 14997 heartbeater.cc:503] Master 127.0.98.62:39857 was elected leader, sending a full tablet report...
I20260620 00:09:22.412884 15139 raft_consensus.cc:348] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ed930b9cba294089a7183326762ac0f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed930b9cba294089a7183326762ac0f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46639 } } peers { permanent_uuid: "5d26247cfb494b33bd2ecbe1494d3ea9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38605 } } peers { permanent_uuid: "ab60a5329cbf4d7995ad7e331de3529d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39455 } }
I20260620 00:09:22.413066 15139 raft_consensus.cc:374] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ed930b9cba294089a7183326762ac0f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:09:22.413101 15139 raft_consensus.cc:729] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ed930b9cba294089a7183326762ac0f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed930b9cba294089a7183326762ac0f9, State: Initialized, Role: FOLLOWER
I20260620 00:09:22.413216 15139 consensus_queue.cc:260] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ed930b9cba294089a7183326762ac0f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed930b9cba294089a7183326762ac0f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46639 } } peers { permanent_uuid: "5d26247cfb494b33bd2ecbe1494d3ea9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38605 } } peers { permanent_uuid: "ab60a5329cbf4d7995ad7e331de3529d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39455 } }
I20260620 00:09:22.413522 15139 ts_tablet_manager.cc:1428] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ed930b9cba294089a7183326762ac0f9: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260620 00:09:22.413578 15129 heartbeater.cc:503] Master 127.0.98.62:39857 was elected leader, sending a full tablet report...
W20260620 00:09:22.506875 14998 tablet.cc:2369] T 5f3c551e427f46ba9a5c01bc97f7fa07 P 5d26247cfb494b33bd2ecbe1494d3ea9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:09:22.583396 14863 raft_consensus.cc:482] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:09:22.583596 14863 raft_consensus.cc:504] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed930b9cba294089a7183326762ac0f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46639 } } peers { permanent_uuid: "5d26247cfb494b33bd2ecbe1494d3ea9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38605 } } peers { permanent_uuid: "ab60a5329cbf4d7995ad7e331de3529d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39455 } }
I20260620 00:09:22.584000 14863 leader_election.cc:288] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed930b9cba294089a7183326762ac0f9 (127.0.98.3:46639), 5d26247cfb494b33bd2ecbe1494d3ea9 (127.0.98.2:38605)
I20260620 00:09:22.588358 15136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:22.584328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45723 (local address 127.0.98.3:46639)
0620 00:09:22.584432 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:22.584434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:22.584462 (+ 28us) server_negotiation.cc:407] Connection header received
0620 00:09:22.584494 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:22.584495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:22.584509 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:22.584581 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:09:22.584828 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:22.585665 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:22.587292 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:22.587933 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:22.588049 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:22.588059 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:22.588235 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 00:09:22.588287 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 00:09:22.588351 14860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:22.584214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46639 (local address 127.0.98.1:45723)
0620 00:09:22.584274 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:22.584286 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:22.584341 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:22.584620 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:22.584622 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:22.584624 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:09:22.584729 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:22.584731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:22.585788 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:22.585788 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:22.587190 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:22.587193 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:22.588070 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:22.588071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:22.588139 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:22.588157 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 00:09:22.588209 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 00:09:22.588265 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":78}
I20260620 00:09:22.588449 15137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:22.584314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37017 (local address 127.0.98.2:38605)
0620 00:09:22.584379 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:22.584381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:22.584511 (+ 130us) server_negotiation.cc:407] Connection header received
0620 00:09:22.584540 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:22.584541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:22.584553 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:22.584610 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:09:22.584816 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:22.585689 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:22.587313 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:22.587942 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:22.588018 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:22.588031 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:22.588325 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 00:09:22.588375 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 00:09:22.588675 15144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:22.584214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38605 (local address 127.0.98.1:37017)
0620 00:09:22.584397 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:22.584405 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:22.584463 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:22.584626 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:22.584627 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:22.584628 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:09:22.584731 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:22.584732 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:22.585780 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:22.585781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:22.587190 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:22.587193 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:22.588080 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:22.588082 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:22.588176 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:22.588179 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:09:22.588293 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 00:09:22.588345 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":128,"thread_start_us":103,"threads_started":1}
I20260620 00:09:22.588798 15083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f3c551e427f46ba9a5c01bc97f7fa07" candidate_uuid: "ab60a5329cbf4d7995ad7e331de3529d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed930b9cba294089a7183326762ac0f9" is_pre_election: true
I20260620 00:09:22.589018 15083 raft_consensus.cc:2454] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ed930b9cba294089a7183326762ac0f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab60a5329cbf4d7995ad7e331de3529d in term 0.
I20260620 00:09:22.589054 14951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f3c551e427f46ba9a5c01bc97f7fa07" candidate_uuid: "ab60a5329cbf4d7995ad7e331de3529d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d26247cfb494b33bd2ecbe1494d3ea9" is_pre_election: true
I20260620 00:09:22.589250 14951 raft_consensus.cc:2454] T 5f3c551e427f46ba9a5c01bc97f7fa07 P 5d26247cfb494b33bd2ecbe1494d3ea9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab60a5329cbf4d7995ad7e331de3529d in term 0.
I20260620 00:09:22.589318 14746 leader_election.cc:302] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab60a5329cbf4d7995ad7e331de3529d, ed930b9cba294089a7183326762ac0f9; no voters:
I20260620 00:09:22.589486 14863 raft_consensus.cc:2781] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:09:22.589555 14863 raft_consensus.cc:482] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:09:22.589576 14863 raft_consensus.cc:3037] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:09:22.590324 14863 raft_consensus.cc:504] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed930b9cba294089a7183326762ac0f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46639 } } peers { permanent_uuid: "5d26247cfb494b33bd2ecbe1494d3ea9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38605 } } peers { permanent_uuid: "ab60a5329cbf4d7995ad7e331de3529d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39455 } }
I20260620 00:09:22.590479 14863 leader_election.cc:288] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [CANDIDATE]: Term 1 election: Requested vote from peers ed930b9cba294089a7183326762ac0f9 (127.0.98.3:46639), 5d26247cfb494b33bd2ecbe1494d3ea9 (127.0.98.2:38605)
I20260620 00:09:22.590652 14951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f3c551e427f46ba9a5c01bc97f7fa07" candidate_uuid: "ab60a5329cbf4d7995ad7e331de3529d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d26247cfb494b33bd2ecbe1494d3ea9"
I20260620 00:09:22.590660 15083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f3c551e427f46ba9a5c01bc97f7fa07" candidate_uuid: "ab60a5329cbf4d7995ad7e331de3529d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed930b9cba294089a7183326762ac0f9"
I20260620 00:09:22.590731 14951 raft_consensus.cc:3037] T 5f3c551e427f46ba9a5c01bc97f7fa07 P 5d26247cfb494b33bd2ecbe1494d3ea9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:09:22.590754 15083 raft_consensus.cc:3037] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ed930b9cba294089a7183326762ac0f9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:09:22.591444 15083 raft_consensus.cc:2454] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ed930b9cba294089a7183326762ac0f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab60a5329cbf4d7995ad7e331de3529d in term 1.
I20260620 00:09:22.591563 14951 raft_consensus.cc:2454] T 5f3c551e427f46ba9a5c01bc97f7fa07 P 5d26247cfb494b33bd2ecbe1494d3ea9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab60a5329cbf4d7995ad7e331de3529d in term 1.
I20260620 00:09:22.591642 14746 leader_election.cc:302] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab60a5329cbf4d7995ad7e331de3529d, ed930b9cba294089a7183326762ac0f9; no voters:
I20260620 00:09:22.591737 14863 raft_consensus.cc:2781] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:09:22.591907 14863 raft_consensus.cc:686] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [term 1 LEADER]: Becoming Leader. State: Replica: ab60a5329cbf4d7995ad7e331de3529d, State: Running, Role: LEADER
I20260620 00:09:22.591988 14863 consensus_queue.cc:237] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed930b9cba294089a7183326762ac0f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46639 } } peers { permanent_uuid: "5d26247cfb494b33bd2ecbe1494d3ea9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38605 } } peers { permanent_uuid: "ab60a5329cbf4d7995ad7e331de3529d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39455 } }
I20260620 00:09:22.592779 14670 catalog_manager.cc:5495] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d reported cstate change: term changed from 0 to 1, leader changed from <none> to ab60a5329cbf4d7995ad7e331de3529d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ab60a5329cbf4d7995ad7e331de3529d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed930b9cba294089a7183326762ac0f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d26247cfb494b33bd2ecbe1494d3ea9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab60a5329cbf4d7995ad7e331de3529d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39455 } health_report { overall_health: HEALTHY } } }
W20260620 00:09:22.616918 14859 tablet.cc:2369] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 00:09:22.628304 15130 tablet.cc:2369] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ed930b9cba294089a7183326762ac0f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:09:22.631595 14866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:22.628758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36976 (local address 127.0.98.1:39455)
0620 00:09:22.628832 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:22.628833 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:22.628846 (+ 13us) server_negotiation.cc:407] Connection header received
0620 00:09:22.628898 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:22.628899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:22.628918 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:22.628970 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:09:22.629129 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:22.629784 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:22.630634 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:22.630863 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:22.630921 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:22.631105 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:09:22.631212 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:09:22.631277 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:22.631418 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 00:09:22.631487 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 00:09:22.633203 14951 raft_consensus.cc:1261] T 5f3c551e427f46ba9a5c01bc97f7fa07 P 5d26247cfb494b33bd2ecbe1494d3ea9 [term 1 FOLLOWER]: Refusing update from remote peer ab60a5329cbf4d7995ad7e331de3529d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:09:22.633221 15083 raft_consensus.cc:1261] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ed930b9cba294089a7183326762ac0f9 [term 1 FOLLOWER]: Refusing update from remote peer ab60a5329cbf4d7995ad7e331de3529d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:09:22.633615 14863 consensus_queue.cc:1035] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d26247cfb494b33bd2ecbe1494d3ea9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:09:22.633805 15145 consensus_queue.cc:1035] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed930b9cba294089a7183326762ac0f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:09:22.637053 15150 mvcc.cc:205] Tried to move back new op lower bound from 7298720410143129600 to 7298720409977319424. Current Snapshot: MvccSnapshot[applied={T|T < 7298720410143129600}]
I20260620 00:09:22.637229 15148 mvcc.cc:205] Tried to move back new op lower bound from 7298720410143129600 to 7298720409977319424. Current Snapshot: MvccSnapshot[applied={T|T < 7298720410143129600 or (T in {7298720410143129600})}]
I20260620 00:09:22.638373 14670 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:35622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:09:22.638487 14670 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:35622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:09:22.639161 14670 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9: Sending DeleteTablet for 3 replicas of tablet 5f3c551e427f46ba9a5c01bc97f7fa07
I20260620 00:09:22.639487 15063 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f3c551e427f46ba9a5c01bc97f7fa07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:09:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38458
I20260620 00:09:22.639498 14931 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f3c551e427f46ba9a5c01bc97f7fa07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:09:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59882
I20260620 00:09:22.639703 14792 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f3c551e427f46ba9a5c01bc97f7fa07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:09:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36968
I20260620 00:09:22.639803 15158 tablet_replica.cc:330] T 5f3c551e427f46ba9a5c01bc97f7fa07 P 5d26247cfb494b33bd2ecbe1494d3ea9: stopping tablet replica
I20260620 00:09:22.639832 14867 tablet_replica.cc:330] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d: stopping tablet replica
I20260620 00:09:22.639905 14867 raft_consensus.cc:2229] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:09:22.639968 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 14708
I20260620 00:09:22.639993 14867 raft_consensus.cc:2258] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:09:22.640676 15158 raft_consensus.cc:2229] T 5f3c551e427f46ba9a5c01bc97f7fa07 P 5d26247cfb494b33bd2ecbe1494d3ea9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:09:22.640831 15157 tablet_replica.cc:330] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ed930b9cba294089a7183326762ac0f9: stopping tablet replica
I20260620 00:09:22.640913 15158 raft_consensus.cc:2258] T 5f3c551e427f46ba9a5c01bc97f7fa07 P 5d26247cfb494b33bd2ecbe1494d3ea9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:09:22.640949 15157 raft_consensus.cc:2229] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ed930b9cba294089a7183326762ac0f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:09:22.641047 15157 raft_consensus.cc:2258] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ed930b9cba294089a7183326762ac0f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:09:22.641561 15158 ts_tablet_manager.cc:1905] T 5f3c551e427f46ba9a5c01bc97f7fa07 P 5d26247cfb494b33bd2ecbe1494d3ea9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:09:22.641568 15157 ts_tablet_manager.cc:1905] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ed930b9cba294089a7183326762ac0f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:09:22.643272 15158 ts_tablet_manager.cc:1918] T 5f3c551e427f46ba9a5c01bc97f7fa07 P 5d26247cfb494b33bd2ecbe1494d3ea9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:09:22.643273 15157 ts_tablet_manager.cc:1918] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ed930b9cba294089a7183326762ac0f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:09:22.643347 15157 log.cc:1196] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ed930b9cba294089a7183326762ac0f9: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/wal/wals/5f3c551e427f46ba9a5c01bc97f7fa07
I20260620 00:09:22.643446 15158 log.cc:1196] T 5f3c551e427f46ba9a5c01bc97f7fa07 P 5d26247cfb494b33bd2ecbe1494d3ea9: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/wal/wals/5f3c551e427f46ba9a5c01bc97f7fa07
I20260620 00:09:22.643659 15157 ts_tablet_manager.cc:1939] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ed930b9cba294089a7183326762ac0f9: Deleting consensus metadata
I20260620 00:09:22.643774 15158 ts_tablet_manager.cc:1939] T 5f3c551e427f46ba9a5c01bc97f7fa07 P 5d26247cfb494b33bd2ecbe1494d3ea9: Deleting consensus metadata
I20260620 00:09:22.644160 14656 catalog_manager.cc:4841] TS ed930b9cba294089a7183326762ac0f9 (127.0.98.3:46639): tablet 5f3c551e427f46ba9a5c01bc97f7fa07 (table test-table [id=d5a043ce560344a095599c5db9583d81]) successfully deleted
I20260620 00:09:22.644346 14654 catalog_manager.cc:4841] TS 5d26247cfb494b33bd2ecbe1494d3ea9 (127.0.98.2:38605): tablet 5f3c551e427f46ba9a5c01bc97f7fa07 (table test-table [id=d5a043ce560344a095599c5db9583d81]) successfully deleted
W20260620 00:09:22.645594 14654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39455 (error 108)
I20260620 00:09:22.645723 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 14864
I20260620 00:09:22.646121 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:22.646015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39455 (local address 127.0.0.1:36990)
0620 00:09:22.646057 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:22.646084 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260620 00:09:22.646363 14654 catalog_manager.cc:4584] TS ab60a5329cbf4d7995ad7e331de3529d (127.0.98.1:39455): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5f3c551e427f46ba9a5c01bc97f7fa07: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39455: connect: Connection refused (error 111)
I20260620 00:09:22.651276 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 15000
I20260620 00:09:22.656490 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 14637
I20260620 00:09:22.662381 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39857
--webserver_interface=127.0.98.62
--webserver_port=41495
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:09:22.742965 15160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:09:22.743232 15160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:09:22.743281 15160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:09:22.744470 15160 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:09:22.744544 15160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:09:22.744570 15160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:09:22.744592 15160 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:09:22.744621 15160 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:09:22.746129 15160 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:09:22.746419 15160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:09:22.746655 15160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:09:22.751664 15166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:22.751637 15168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:22.751627 15165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:09:22.751698 15160 server_base.cc:1034] running on GCE node
I20260620 00:09:22.752089 15160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:09:22.752385 15160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:09:22.753535 15160 hybrid_clock.cc:648] HybridClock initialized: now 1781914162753515 us; error 40 us; skew 500 ppm
Jun 20 00:09:22 dist-test-slave-grqt krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:09:22.756074 15160 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:09:22.756511 15160 webserver.cc:460] Webserver started at http://127.0.98.62:41495/ using document root <none> and password file <none>
I20260620 00:09:22.756734 15160 fs_manager.cc:359] Metadata directory not provided
I20260620 00:09:22.756803 15160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:09:22.758214 15160 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:09:22.758811 15175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:22.759001 15160 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 00:09:22.759068 15160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "97eabdebe29c4620b9981c6b4e3592e9"
format_stamp: "Formatted at 2026-06-20 00:09:21 on dist-test-slave-grqt"
server_key: "00e0c4144e4770b9adba18c0a6558286"
server_key_iv: "64768d8a24eb34a7d8cd44a62787f941"
server_key_version: "encryptionkey@0"
I20260620 00:09:22.759393 15160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:09:22.777940 15160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:09:22.780194 15160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:09:22.780315 15160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:09:22.784776 15160 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39857
I20260620 00:09:22.784811 15227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39857 every 8 connection(s)
I20260620 00:09:22.785228 15160 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:09:22.786922 15228 sys_catalog.cc:263] Verifying existing consensus state
I20260620 00:09:22.787421 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 15160
I20260620 00:09:22.787688 15228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9: Bootstrap starting.
I20260620 00:09:22.787709 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39455
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39857
--webserver_port=45377
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:09:22.796051 15228 log.cc:824] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9: Log is configured to *not* fsync() on all Append() calls
I20260620 00:09:22.798462 15228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 00:09:22.798784 15228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9: Bootstrap complete.
I20260620 00:09:22.801023 15228 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97eabdebe29c4620b9981c6b4e3592e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39857 } }
I20260620 00:09:22.801288 15228 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97eabdebe29c4620b9981c6b4e3592e9, State: Initialized, Role: FOLLOWER
I20260620 00:09:22.801522 15228 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97eabdebe29c4620b9981c6b4e3592e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39857 } }
I20260620 00:09:22.801620 15228 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:09:22.801672 15228 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:09:22.801721 15228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 00:09:22.802701 15228 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97eabdebe29c4620b9981c6b4e3592e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39857 } }
I20260620 00:09:22.802834 15228 leader_election.cc:302] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97eabdebe29c4620b9981c6b4e3592e9; no voters:
I20260620 00:09:22.803040 15228 leader_election.cc:288] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 00:09:22.803079 15232 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 00:09:22.803191 15232 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [term 3 LEADER]: Becoming Leader. State: Replica: 97eabdebe29c4620b9981c6b4e3592e9, State: Running, Role: LEADER
I20260620 00:09:22.803284 15232 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97eabdebe29c4620b9981c6b4e3592e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39857 } }
I20260620 00:09:22.803313 15228 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:09:22.803577 15234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97eabdebe29c4620b9981c6b4e3592e9. Latest consensus state: current_term: 3 leader_uuid: "97eabdebe29c4620b9981c6b4e3592e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97eabdebe29c4620b9981c6b4e3592e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39857 } } }
I20260620 00:09:22.803654 15234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [sys.catalog]: This master's current role is: LEADER
I20260620 00:09:22.803748 15233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "97eabdebe29c4620b9981c6b4e3592e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97eabdebe29c4620b9981c6b4e3592e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39857 } } }
I20260620 00:09:22.803884 15233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97eabdebe29c4620b9981c6b4e3592e9 [sys.catalog]: This master's current role is: LEADER
I20260620 00:09:22.803913 15236 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:09:22.805352 15236 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:09:22.805487 15236 catalog_manager.cc:1259] Loaded cluster ID: 4b1958d88f7b469eb7dd03fa0979b716
I20260620 00:09:22.805518 15236 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 00:09:22.805948 15236 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa06886b9ad google::LogMessage::Fail() at ??:0
@ 0x7fa06886e88a google::LogMessage::SendToLog() at ??:0
@ 0x7fa06886b4de google::LogMessage::Flush() at ??:0
@ 0x7fa06886ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa0707900f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fa0707998b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa06a7390d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa06a72e954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa06b5926db start_thread at ??:0
@ 0x7fa06736961f clone at ??:0
W20260620 00:09:22.885407 15230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:09:22.885619 15230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:09:22.885644 15230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:09:22.886888 15230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:09:22.886942 15230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:09:22.886971 15230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:09:22.888494 15230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39455
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45377
--tserver_master_addrs=127.0.98.62:39857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:09:22.888751 15230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:09:22.889012 15230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:09:22.894235 15267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:09:22.894344 15230 server_base.cc:1034] running on GCE node
W20260620 00:09:22.894228 15264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:22.894228 15265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:09:22.894742 15230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:09:22.895016 15230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:09:22.896183 15230 hybrid_clock.cc:648] HybridClock initialized: now 1781914162896186 us; error 46 us; skew 500 ppm
Jun 20 00:09:22 dist-test-slave-grqt krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:09:22.899019 15230 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:09:22.899492 15230 webserver.cc:460] Webserver started at http://127.0.98.1:45377/ using document root <none> and password file <none>
I20260620 00:09:22.899724 15230 fs_manager.cc:359] Metadata directory not provided
I20260620 00:09:22.899801 15230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:09:22.901283 15230 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:09:22.902076 15274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:22.902312 15230 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:09:22.902428 15230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/wal
uuid: "ab60a5329cbf4d7995ad7e331de3529d"
format_stamp: "Formatted at 2026-06-20 00:09:21 on dist-test-slave-grqt"
server_key: "18d71d70ac1afa913059b328807c0be7"
server_key_iv: "bc6bdfa337a99afc268ef29bfdb7eaf2"
server_key_version: "encryptionkey@0"
I20260620 00:09:22.902837 15230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:09:22.921644 15230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:09:22.924397 15230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:09:22.924556 15230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:09:22.924813 15230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:09:22.925346 15281 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 00:09:22.926908 15230 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 00:09:22.926975 15230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 00:09:22.927011 15230 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 00:09:22.927901 15230 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 00:09:22.927948 15230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 00:09:22.928052 15281 tablet_bootstrap.cc:492] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d: Bootstrap starting.
I20260620 00:09:22.935529 15230 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39455
I20260620 00:09:22.935726 15388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39455 every 8 connection(s)
I20260620 00:09:22.935983 15230 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:09:22.938669 15281 log.cc:824] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d: Log is configured to *not* fsync() on all Append() calls
I20260620 00:09:22.940856 15281 tablet_bootstrap.cc:492] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 00:09:22.941143 15281 tablet_bootstrap.cc:492] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d: Bootstrap complete.
I20260620 00:09:22.942190 15281 ts_tablet_manager.cc:1397] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.010s
I20260620 00:09:22.943117 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 15230
I20260620 00:09:22.943398 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38605
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39857
--webserver_port=36799
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:09:22.943500 15281 raft_consensus.cc:348] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed930b9cba294089a7183326762ac0f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46639 } } peers { permanent_uuid: "5d26247cfb494b33bd2ecbe1494d3ea9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38605 } } peers { permanent_uuid: "ab60a5329cbf4d7995ad7e331de3529d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39455 } }
I20260620 00:09:22.943907 15281 raft_consensus.cc:729] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab60a5329cbf4d7995ad7e331de3529d, State: Initialized, Role: FOLLOWER
I20260620 00:09:22.944077 15281 consensus_queue.cc:260] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed930b9cba294089a7183326762ac0f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46639 } } peers { permanent_uuid: "5d26247cfb494b33bd2ecbe1494d3ea9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38605 } } peers { permanent_uuid: "ab60a5329cbf4d7995ad7e331de3529d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39455 } }
I20260620 00:09:22.944365 15281 ts_tablet_manager.cc:1428] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260620 00:09:23.014191 15391 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:09:22.936703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.1:55737)
0620 00:09:22.936924 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:22.936979 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:22.937770 (+ 791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:23.013976 (+ 76206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":96,"threads_started":1}
W20260620 00:09:23.014478 15389 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39857 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39857: Client connection negotiation failed: client connection to 127.0.98.62:39857: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 00:09:23.014817 15391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:23.014690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.1:55801)
0620 00:09:23.014746 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:23.014793 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 00:09:23.015228 15391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:23.015103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.1:41817)
0620 00:09:23.015174 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:23.015206 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 00:09:23.015416 15389 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 00:09:23.029083 15394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:09:23.029292 15394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:09:23.029316 15394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:09:23.030539 15394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:09:23.030583 15394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:09:23.030609 15394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:09:23.032080 15394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38605
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36799
--tserver_master_addrs=127.0.98.62:39857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:09:23.032322 15394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:09:23.032548 15394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:09:23.037663 15400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:09:23.037741 15394 server_base.cc:1034] running on GCE node
W20260620 00:09:23.037649 15399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:23.037665 15402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:09:23.038059 15394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:09:23.038256 15394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:09:23.039443 15394 hybrid_clock.cc:648] HybridClock initialized: now 1781914163039436 us; error 40 us; skew 500 ppm
Jun 20 00:09:23 dist-test-slave-grqt krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:09:23.042567 15394 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:09:23.043020 15394 webserver.cc:460] Webserver started at http://127.0.98.2:36799/ using document root <none> and password file <none>
I20260620 00:09:23.043200 15394 fs_manager.cc:359] Metadata directory not provided
I20260620 00:09:23.043257 15394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:09:23.044535 15394 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:09:23.045132 15409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:23.045326 15394 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:09:23.045437 15394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/wal
uuid: "5d26247cfb494b33bd2ecbe1494d3ea9"
format_stamp: "Formatted at 2026-06-20 00:09:21 on dist-test-slave-grqt"
server_key: "57f2e713c9b24fbb66ef98a1b8cb90af"
server_key_iv: "1db291533152684dc6d9581b19a58b58"
server_key_version: "encryptionkey@0"
I20260620 00:09:23.045783 15394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:09:23.057636 15394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:09:23.059798 15394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:09:23.059976 15394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:09:23.060220 15394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:09:23.060530 15394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:09:23.060585 15394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:23.060628 15394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:09:23.060655 15394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:23.066716 15394 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38605
I20260620 00:09:23.066790 15522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38605 every 8 connection(s)
I20260620 00:09:23.067153 15394 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:09:23.068291 15525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:23.067909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.2:59003)
0620 00:09:23.068072 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:23.068142 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
W20260620 00:09:23.068730 15523 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39857 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39857: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
I20260620 00:09:23.069051 15525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:23.068946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.2:59303)
0620 00:09:23.068998 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:23.069031 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 00:09:23.069162 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 15394
I20260620 00:09:23.069427 15525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:23.069277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.2:34257)
0620 00:09:23.069336 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:23.069402 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 00:09:23.069511 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46639
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39857
--webserver_port=39401
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:09:23.069685 15523 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 00:09:23.153520 15526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:09:23.153724 15526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:09:23.153774 15526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:09:23.155006 15526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:09:23.155048 15526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:09:23.155074 15526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:09:23.156518 15526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46639
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39401
--tserver_master_addrs=127.0.98.62:39857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:09:23.156708 15526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:09:23.156894 15526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 00:09:23.161933 15526 server_base.cc:1034] running on GCE node
W20260620 00:09:23.161973 15532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:23.162065 15534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:23.161940 15531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:09:23.162379 15526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:09:23.162642 15526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:09:23.163813 15526 hybrid_clock.cc:648] HybridClock initialized: now 1781914163163807 us; error 37 us; skew 500 ppm
Jun 20 00:09:23 dist-test-slave-grqt krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:09:23.166503 15526 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:09:23.166966 15526 webserver.cc:460] Webserver started at http://127.0.98.3:39401/ using document root <none> and password file <none>
I20260620 00:09:23.167203 15526 fs_manager.cc:359] Metadata directory not provided
I20260620 00:09:23.167279 15526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:09:23.168618 15526 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:09:23.169220 15541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:23.169440 15526 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:09:23.169536 15526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/wal
uuid: "ed930b9cba294089a7183326762ac0f9"
format_stamp: "Formatted at 2026-06-20 00:09:21 on dist-test-slave-grqt"
server_key: "f3c2ede53150f0847211bae243c9f748"
server_key_iv: "2aab5a610bd19d23eb3f821cc107fb3f"
server_key_version: "encryptionkey@0"
I20260620 00:09:23.169922 15526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:09:23.182466 15526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:09:23.184334 15526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:09:23.184501 15526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:09:23.184734 15526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:09:23.185050 15526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:09:23.185104 15526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:23.185145 15526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:09:23.185173 15526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:23.190928 15526 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46639
I20260620 00:09:23.190999 15654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46639 every 8 connection(s)
I20260620 00:09:23.191356 15526 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:09:23.192459 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:23.192056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.3:59579)
0620 00:09:23.192240 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:23.192316 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":77,"threads_started":1}
W20260620 00:09:23.192884 15655 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39857 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39857: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
I20260620 00:09:23.193156 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:23.193085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.3:57215)
0620 00:09:23.193115 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:23.193139 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260620 00:09:23.193410 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:23.193340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.3:43693)
0620 00:09:23.193361 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:23.193397 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260620 00:09:23.193533 15655 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 00:09:23.194414 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 15526
I20260620 00:09:24.016114 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:24.015841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.1:38443)
0620 00:09:24.016002 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:24.016051 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":67,"threads_started":1}
I20260620 00:09:24.070228 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:24.070031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.2:56847)
0620 00:09:24.070144 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:24.070181 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":53,"threads_started":1}
I20260620 00:09:24.194084 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:24.193847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.3:44093)
0620 00:09:24.193994 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:24.194033 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":95,"threads_started":1}
I20260620 00:09:24.617607 15662 raft_consensus.cc:482] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:09:24.617730 15662 raft_consensus.cc:504] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed930b9cba294089a7183326762ac0f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46639 } } peers { permanent_uuid: "5d26247cfb494b33bd2ecbe1494d3ea9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38605 } } peers { permanent_uuid: "ab60a5329cbf4d7995ad7e331de3529d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39455 } }
I20260620 00:09:24.618080 15662 leader_election.cc:288] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ed930b9cba294089a7183326762ac0f9 (127.0.98.3:46639), 5d26247cfb494b33bd2ecbe1494d3ea9 (127.0.98.2:38605)
Jun 20 00:09:24 dist-test-slave-grqt krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Jun 20 00:09:24 dist-test-slave-grqt krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260620 00:09:24.629666 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:24.618396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47609 (local address 127.0.98.2:38605)
0620 00:09:24.618543 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:24.618556 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:24.618582 (+ 26us) server_negotiation.cc:407] Connection header received
0620 00:09:24.619056 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:24.619058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:24.619101 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:24.619211 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:09:24.619974 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:24.620855 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:24.621975 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:24.622395 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:24.622523 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:24.625151 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:09:24.625156 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:09:24.625173 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:09:24.625188 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:09:24.627447 (+ 2259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:24.627993 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:24.627997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:24.627997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:24.628094 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:24.628520 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:24.628522 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:24.628522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:24.628866 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:09:24.628967 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:24.629363 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 00:09:24.629543 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":46,"threads_started":1}
I20260620 00:09:24.629666 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:24.618481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39897 (local address 127.0.98.3:46639)
0620 00:09:24.618645 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:24.618654 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:24.618697 (+ 43us) server_negotiation.cc:407] Connection header received
0620 00:09:24.619097 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:24.619099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:24.619134 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:24.619291 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:09:24.619992 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:24.620844 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:24.622012 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:24.622397 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:24.622522 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:24.625412 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:09:24.625416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:09:24.625416 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:09:24.625434 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:09:24.627295 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:24.628190 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:24.628193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:24.628193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:24.628274 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:24.628520 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:24.628522 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:24.628522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:24.628866 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:09:24.628963 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:24.629363 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 00:09:24.629554 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":67,"threads_started":1}
I20260620 00:09:24.629673 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:24.618284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38605 (local address 127.0.98.1:47609)
0620 00:09:24.618441 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:24.618454 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:24.618514 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:24.619290 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:24.619293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:24.619664 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:09:24.619844 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:24.619850 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:24.620974 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:24.620975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:24.621884 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:24.621887 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:24.622491 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:24.622492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:24.622612 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:24.623291 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:09:24.623295 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:09:24.624855 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:09:24.627526 (+ 2671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:24.627527 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:24.627527 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:24.627891 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:24.628183 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:24.628184 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:24.628184 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:24.628388 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:24.628971 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:09:24.628981 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:09:24.629149 (+ 168us) client_negotiation.cc:769] Sending connection context
0620 00:09:24.629302 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 00:09:24.629547 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":53,"thread_start_us":78,"threads_started":1}
I20260620 00:09:24.629726 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:24.618285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46639 (local address 127.0.98.1:39897)
0620 00:09:24.618599 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:24.618609 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:24.618665 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:24.619293 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:24.619293 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:24.619664 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:09:24.619844 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:24.619850 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:24.620948 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:24.620949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:24.621903 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:24.621906 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:24.622492 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:24.622493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:24.622614 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:24.623226 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:09:24.623235 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:09:24.625272 (+ 2037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:09:24.627435 (+ 2163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:24.627442 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:24.627445 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:24.627891 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:24.628354 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:24.628355 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:24.628355 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:24.628423 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:24.628971 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:09:24.628981 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:09:24.629149 (+ 168us) client_negotiation.cc:769] Sending connection context
0620 00:09:24.629276 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 00:09:24.629605 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":181,"thread_start_us":88,"threads_started":1}
W20260620 00:09:24.630563 15277 leader_election.cc:341] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ed930b9cba294089a7183326762ac0f9 (127.0.98.3:46639): Not found: Tablet not found: 5f3c551e427f46ba9a5c01bc97f7fa07
W20260620 00:09:24.630679 15275 leader_election.cc:341] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5d26247cfb494b33bd2ecbe1494d3ea9 (127.0.98.2:38605): Not found: Tablet not found: 5f3c551e427f46ba9a5c01bc97f7fa07
I20260620 00:09:24.630743 15275 leader_election.cc:302] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab60a5329cbf4d7995ad7e331de3529d; no voters: 5d26247cfb494b33bd2ecbe1494d3ea9, ed930b9cba294089a7183326762ac0f9
I20260620 00:09:24.630882 15662 raft_consensus.cc:2726] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 00:09:25.017011 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:25.016750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.1:34051)
0620 00:09:25.016875 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:25.016920 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260620 00:09:25.070974 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:25.070727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.2:55411)
0620 00:09:25.070870 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:25.070918 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":65,"threads_started":1}
I20260620 00:09:25.194738 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:25.194547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.3:35391)
0620 00:09:25.194662 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:25.194701 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
I20260620 00:09:26.017802 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:26.017557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.1:58239)
0620 00:09:26.017692 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:26.017738 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":64,"threads_started":1}
I20260620 00:09:26.071911 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:26.071570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.2:44543)
0620 00:09:26.071775 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:26.071826 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":76,"threads_started":1}
I20260620 00:09:26.195410 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:26.195169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.3:37191)
0620 00:09:26.195322 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:26.195360 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":74,"threads_started":1}
I20260620 00:09:26.651657 15672 raft_consensus.cc:482] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:09:26.651772 15672 raft_consensus.cc:504] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed930b9cba294089a7183326762ac0f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46639 } } peers { permanent_uuid: "5d26247cfb494b33bd2ecbe1494d3ea9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38605 } } peers { permanent_uuid: "ab60a5329cbf4d7995ad7e331de3529d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39455 } }
I20260620 00:09:26.651983 15672 leader_election.cc:288] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ed930b9cba294089a7183326762ac0f9 (127.0.98.3:46639), 5d26247cfb494b33bd2ecbe1494d3ea9 (127.0.98.2:38605)
W20260620 00:09:26.652470 15277 leader_election.cc:341] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ed930b9cba294089a7183326762ac0f9 (127.0.98.3:46639): Not found: Tablet not found: 5f3c551e427f46ba9a5c01bc97f7fa07
W20260620 00:09:26.652542 15275 leader_election.cc:341] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5d26247cfb494b33bd2ecbe1494d3ea9 (127.0.98.2:38605): Not found: Tablet not found: 5f3c551e427f46ba9a5c01bc97f7fa07
I20260620 00:09:26.652637 15275 leader_election.cc:302] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab60a5329cbf4d7995ad7e331de3529d; no voters: 5d26247cfb494b33bd2ecbe1494d3ea9, ed930b9cba294089a7183326762ac0f9
I20260620 00:09:26.652735 15672 raft_consensus.cc:2726] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 00:09:27.018709 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:27.018438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.1:55455)
0620 00:09:27.018609 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:27.018660 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":68,"threads_started":1}
I20260620 00:09:27.072747 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:27.072465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.2:54165)
0620 00:09:27.072610 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:27.072656 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":64,"threads_started":1}
I20260620 00:09:27.196143 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:27.195893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.3:39829)
0620 00:09:27.196060 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:27.196103 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":92,"threads_started":1}
I20260620 00:09:28.019550 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:28.019313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.1:42347)
0620 00:09:28.019458 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:28.019504 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":85,"threads_started":1}
I20260620 00:09:28.073721 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:28.073401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.2:46535)
0620 00:09:28.073613 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:28.073666 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":75,"threads_started":1}
I20260620 00:09:28.196864 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:28.196606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.3:57021)
0620 00:09:28.196768 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:28.196818 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":75,"threads_started":1}
I20260620 00:09:28.545862 15679 raft_consensus.cc:482] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:09:28.545948 15679 raft_consensus.cc:504] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed930b9cba294089a7183326762ac0f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46639 } } peers { permanent_uuid: "5d26247cfb494b33bd2ecbe1494d3ea9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38605 } } peers { permanent_uuid: "ab60a5329cbf4d7995ad7e331de3529d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39455 } }
I20260620 00:09:28.546144 15679 leader_election.cc:288] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ed930b9cba294089a7183326762ac0f9 (127.0.98.3:46639), 5d26247cfb494b33bd2ecbe1494d3ea9 (127.0.98.2:38605)
W20260620 00:09:28.546517 15275 leader_election.cc:341] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5d26247cfb494b33bd2ecbe1494d3ea9 (127.0.98.2:38605): Not found: Tablet not found: 5f3c551e427f46ba9a5c01bc97f7fa07
W20260620 00:09:28.546615 15277 leader_election.cc:341] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ed930b9cba294089a7183326762ac0f9 (127.0.98.3:46639): Not found: Tablet not found: 5f3c551e427f46ba9a5c01bc97f7fa07
I20260620 00:09:28.546696 15277 leader_election.cc:302] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab60a5329cbf4d7995ad7e331de3529d; no voters: 5d26247cfb494b33bd2ecbe1494d3ea9, ed930b9cba294089a7183326762ac0f9
I20260620 00:09:28.546795 15679 raft_consensus.cc:2726] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 00:09:29.020354 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:29.020058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.1:34801)
0620 00:09:29.020195 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:29.020240 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":75,"threads_started":1}
I20260620 00:09:29.074368 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:29.074153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.2:38565)
0620 00:09:29.074276 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:29.074320 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
I20260620 00:09:29.197598 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:29.197343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.3:45129)
0620 00:09:29.197513 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:29.197554 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":123,"threads_started":1}
I20260620 00:09:30.021234 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:30.020979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.1:59065)
0620 00:09:30.021142 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:30.021188 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":90,"threads_started":1}
I20260620 00:09:30.075167 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:30.074930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.2:58243)
0620 00:09:30.075076 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:30.075121 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":60,"threads_started":1}
I20260620 00:09:30.198323 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:30.198098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.3:56413)
0620 00:09:30.198241 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:30.198278 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
I20260620 00:09:31.022089 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:31.021860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.1:36419)
0620 00:09:31.022008 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:31.022044 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":89,"threads_started":1}
I20260620 00:09:31.039661 15687 raft_consensus.cc:482] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:09:31.039765 15687 raft_consensus.cc:504] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed930b9cba294089a7183326762ac0f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46639 } } peers { permanent_uuid: "5d26247cfb494b33bd2ecbe1494d3ea9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38605 } } peers { permanent_uuid: "ab60a5329cbf4d7995ad7e331de3529d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39455 } }
I20260620 00:09:31.040000 15687 leader_election.cc:288] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ed930b9cba294089a7183326762ac0f9 (127.0.98.3:46639), 5d26247cfb494b33bd2ecbe1494d3ea9 (127.0.98.2:38605)
W20260620 00:09:31.040439 15275 leader_election.cc:341] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5d26247cfb494b33bd2ecbe1494d3ea9 (127.0.98.2:38605): Not found: Tablet not found: 5f3c551e427f46ba9a5c01bc97f7fa07
W20260620 00:09:31.040541 15277 leader_election.cc:341] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ed930b9cba294089a7183326762ac0f9 (127.0.98.3:46639): Not found: Tablet not found: 5f3c551e427f46ba9a5c01bc97f7fa07
I20260620 00:09:31.040603 15277 leader_election.cc:302] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab60a5329cbf4d7995ad7e331de3529d; no voters: 5d26247cfb494b33bd2ecbe1494d3ea9, ed930b9cba294089a7183326762ac0f9
I20260620 00:09:31.040664 15687 raft_consensus.cc:2726] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 00:09:31.075949 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:31.075693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.2:38523)
0620 00:09:31.075845 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:31.075887 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":81,"threads_started":1}
I20260620 00:09:31.199043 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:31.198809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.3:43099)
0620 00:09:31.198924 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:31.198967 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":58,"threads_started":1}
I20260620 00:09:32.022912 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:32.022691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.1:38685)
0620 00:09:32.022824 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:32.022865 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":66,"threads_started":1}
I20260620 00:09:32.076846 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:32.076620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.2:47177)
0620 00:09:32.076767 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:32.076802 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":85,"threads_started":1}
I20260620 00:09:32.199803 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:32.199556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.3:59059)
0620 00:09:32.199706 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:32.199752 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":63,"threads_started":1}
I20260620 00:09:33.023562 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:33.023359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.1:36241)
0620 00:09:33.023481 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:33.023514 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":60,"threads_started":1}
I20260620 00:09:33.077507 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:33.077265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.2:34201)
0620 00:09:33.077419 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:33.077460 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":66,"threads_started":1}
I20260620 00:09:33.200659 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:33.200390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.3:43513)
0620 00:09:33.200557 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:33.200607 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":71,"threads_started":1}
I20260620 00:09:33.929082 15696 raft_consensus.cc:482] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:09:33.929208 15696 raft_consensus.cc:504] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed930b9cba294089a7183326762ac0f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46639 } } peers { permanent_uuid: "5d26247cfb494b33bd2ecbe1494d3ea9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38605 } } peers { permanent_uuid: "ab60a5329cbf4d7995ad7e331de3529d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39455 } }
I20260620 00:09:33.929416 15696 leader_election.cc:288] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ed930b9cba294089a7183326762ac0f9 (127.0.98.3:46639), 5d26247cfb494b33bd2ecbe1494d3ea9 (127.0.98.2:38605)
W20260620 00:09:33.929790 15277 leader_election.cc:341] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ed930b9cba294089a7183326762ac0f9 (127.0.98.3:46639): Not found: Tablet not found: 5f3c551e427f46ba9a5c01bc97f7fa07
W20260620 00:09:33.929899 15275 leader_election.cc:341] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5d26247cfb494b33bd2ecbe1494d3ea9 (127.0.98.2:38605): Not found: Tablet not found: 5f3c551e427f46ba9a5c01bc97f7fa07
I20260620 00:09:33.929981 15275 leader_election.cc:302] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab60a5329cbf4d7995ad7e331de3529d; no voters: 5d26247cfb494b33bd2ecbe1494d3ea9, ed930b9cba294089a7183326762ac0f9
I20260620 00:09:33.930080 15696 raft_consensus.cc:2726] T 5f3c551e427f46ba9a5c01bc97f7fa07 P ab60a5329cbf4d7995ad7e331de3529d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 00:09:34.024322 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:34.024074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.1:40949)
0620 00:09:34.024224 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:34.024262 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":100,"threads_started":1}
I20260620 00:09:34.078282 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:34.078033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.2:55717)
0620 00:09:34.078174 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:34.078235 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
I20260620 00:09:34.201552 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:34.201249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.3:55183)
0620 00:09:34.201455 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:34.201501 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":91,"threads_started":1}
I20260620 00:09:35.025152 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:35.024901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.1:46591)
0620 00:09:35.025062 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:35.025107 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":68,"threads_started":1}
I20260620 00:09:35.079123 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:35.078849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.2:49369)
0620 00:09:35.079020 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:35.079069 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":74,"threads_started":1}
I20260620 00:09:35.202268 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:35.202050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.3:53363)
0620 00:09:35.202175 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:35.202218 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":59,"threads_started":1}
I20260620 00:09:36.025962 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:36.025717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.1:36939)
0620 00:09:36.025868 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:36.025916 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":68,"threads_started":1}
I20260620 00:09:36.079981 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:36.079702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.2:41175)
0620 00:09:36.079859 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:36.079933 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":68,"threads_started":1}
I20260620 00:09:36.203094 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:36.202808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.3:44039)
0620 00:09:36.202996 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:36.203045 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":100,"threads_started":1}
I20260620 00:09:37.026772 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:37.026546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.1:54629)
0620 00:09:37.026679 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:37.026726 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":67,"threads_started":1}
I20260620 00:09:37.080855 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:37.080606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.2:35599)
0620 00:09:37.080751 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:37.080808 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":67,"threads_started":1}
I20260620 00:09:37.203881 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:37.203652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.3:47209)
0620 00:09:37.203795 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:37.203833 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":79,"threads_started":1}
I20260620 00:09:38.027684 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:38.027414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.1:38285)
0620 00:09:38.027593 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:38.027638 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":70,"threads_started":1}
I20260620 00:09:38.081866 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:38.081630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.2:53203)
0620 00:09:38.081755 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:38.081803 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
I20260620 00:09:38.198627 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 15230
I20260620 00:09:38.203897 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 15394
I20260620 00:09:38.205760 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:38.205524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.3:46551)
0620 00:09:38.205666 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:38.205713 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
I20260620 00:09:38.208606 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 15526
2026-06-20T00:09:38Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18374 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:09:38 dist-test-slave-grqt krb5kdc[15714](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:09:38 dist-test-slave-grqt krb5kdc[15714](info): set up 2 sockets
Jun 20 00:09:38 dist-test-slave-grqt krb5kdc[15714](info): commencing operation
krb5kdc: starting...
W20260620 00:09:39.265407 392 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:09:39 dist-test-slave-grqt krb5kdc[15714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:09:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:09:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:09:39.389119 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33689
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:09:39.470396 15727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:09:39.470654 15727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:09:39.470692 15727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:09:39.471879 15727 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 00:09:39.471956 15727 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 00:09:39.471980 15727 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 00:09:39.471999 15727 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:09:39.472030 15727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:09:39.472059 15727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:09:39.472086 15727 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:09:39.472107 15727 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:09:39.473670 15727 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:09:39.473963 15727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:09:39.474192 15727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:09:39.479540 15735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:39.479523 15733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:09:39.479676 15727 server_base.cc:1034] running on GCE node
W20260620 00:09:39.479678 15732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:09:39.480158 15727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:09:39.480485 15727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:09:39.481662 15727 hybrid_clock.cc:648] HybridClock initialized: now 1781914179481667 us; error 52 us; skew 500 ppm
Jun 20 00:09:39 dist-test-slave-grqt krb5kdc[15714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:09:39.484355 15727 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:09:39.484838 15727 webserver.cc:460] Webserver started at http://127.0.98.62:40669/ using document root <none> and password file <none>
I20260620 00:09:39.485081 15727 fs_manager.cc:359] Metadata directory not provided
I20260620 00:09:39.485127 15727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:09:39.485275 15727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:09:39.486410 15727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "136468519525499ab7998577528f32b0"
format_stamp: "Formatted at 2026-06-20 00:09:39 on dist-test-slave-grqt"
server_key: "65b8edbc516ba838cdcc3be23b7bf168"
server_key_iv: "ea93e8fb0e8c9ee5b9edf39a19898c7f"
server_key_version: "encryptionkey@0"
I20260620 00:09:39.486776 15727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "136468519525499ab7998577528f32b0"
format_stamp: "Formatted at 2026-06-20 00:09:39 on dist-test-slave-grqt"
server_key: "65b8edbc516ba838cdcc3be23b7bf168"
server_key_iv: "ea93e8fb0e8c9ee5b9edf39a19898c7f"
server_key_version: "encryptionkey@0"
I20260620 00:09:39.488145 15727 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:09:39.488950 15742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:39.489148 15727 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:09:39.489269 15727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "136468519525499ab7998577528f32b0"
format_stamp: "Formatted at 2026-06-20 00:09:39 on dist-test-slave-grqt"
server_key: "65b8edbc516ba838cdcc3be23b7bf168"
server_key_iv: "ea93e8fb0e8c9ee5b9edf39a19898c7f"
server_key_version: "encryptionkey@0"
I20260620 00:09:39.489352 15727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:09:39.517719 15727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:09:39.520239 15727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:09:39.520413 15727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:09:39.524533 15727 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33689
I20260620 00:09:39.524576 15794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33689 every 8 connection(s)
I20260620 00:09:39.524960 15727 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:09:39.525887 15795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:09:39.528174 15795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0: Bootstrap starting.
I20260620 00:09:39.528772 15795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0: Neither blocks nor log segments found. Creating new log.
I20260620 00:09:39.529074 15795 log.cc:824] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0: Log is configured to *not* fsync() on all Append() calls
I20260620 00:09:39.529838 15795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0: No bootstrap required, opened a new log
I20260620 00:09:39.531390 15795 raft_consensus.cc:348] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136468519525499ab7998577528f32b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33689 } }
I20260620 00:09:39.531533 15795 raft_consensus.cc:374] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:09:39.531558 15795 raft_consensus.cc:729] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136468519525499ab7998577528f32b0, State: Initialized, Role: FOLLOWER
I20260620 00:09:39.531736 15795 consensus_queue.cc:260] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136468519525499ab7998577528f32b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33689 } }
I20260620 00:09:39.531801 15795 raft_consensus.cc:388] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:09:39.531822 15795 raft_consensus.cc:482] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:09:39.531891 15795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:09:39.532428 15795 raft_consensus.cc:504] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136468519525499ab7998577528f32b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33689 } }
I20260620 00:09:39.532538 15795 leader_election.cc:302] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 136468519525499ab7998577528f32b0; no voters:
I20260620 00:09:39.532742 15795 leader_election.cc:288] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:09:39.532811 15798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:09:39.532984 15795 sys_catalog.cc:564] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:09:39.532982 15798 raft_consensus.cc:686] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [term 1 LEADER]: Becoming Leader. State: Replica: 136468519525499ab7998577528f32b0, State: Running, Role: LEADER
I20260620 00:09:39.533108 15798 consensus_queue.cc:237] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136468519525499ab7998577528f32b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33689 } }
I20260620 00:09:39.533612 15800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 136468519525499ab7998577528f32b0. Latest consensus state: current_term: 1 leader_uuid: "136468519525499ab7998577528f32b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136468519525499ab7998577528f32b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33689 } } }
I20260620 00:09:39.533622 15799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "136468519525499ab7998577528f32b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136468519525499ab7998577528f32b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33689 } } }
I20260620 00:09:39.533721 15800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [sys.catalog]: This master's current role is: LEADER
I20260620 00:09:39.533726 15799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [sys.catalog]: This master's current role is: LEADER
I20260620 00:09:39.533926 15806 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:09:39.534415 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 15727
I20260620 00:09:39.534459 15806 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:09:39.534514 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:09:39.534775 392 external_mini_cluster.cc:1428] Setting key 4f92c7967b418212e7e611c81151db42
I20260620 00:09:39.536149 15806 catalog_manager.cc:1347] Generated new cluster ID: 6de8495070104f8ea90f0bd0a936c268
I20260620 00:09:39.536198 15806 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 00:09:39 dist-test-slave-grqt krb5kdc[15714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:09:39.548795 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:39.535234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44366 (local address 127.0.98.62:33689)
0620 00:09:39.537554 (+ 2320us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:39.537566 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:39.537602 (+ 36us) server_negotiation.cc:407] Connection header received
0620 00:09:39.537731 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:39.537738 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:39.537792 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:39.537938 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:09:39.538714 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:39.540206 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:39.541241 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:39.541968 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:39.542099 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:39.544963 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:09:39.544968 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:09:39.544968 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:09:39.544981 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:09:39.547046 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:39.547480 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:39.547483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:39.547484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:39.547602 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:39.547822 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:39.547823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:39.547823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:39.548129 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:09:39.548247 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:39.548404 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 00:09:39.548541 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2264,"thread_start_us":75,"threads_started":1}
I20260620 00:09:39.558120 15806 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:09:39.558883 15806 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:09:39.562496 15806 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0: Generated new TSK 0
I20260620 00:09:39.562677 15806 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:09:39.592437 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33689
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:09:39.674821 15821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:09:39.675047 15821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:09:39.675071 15821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:09:39.676168 15821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:09:39.676203 15821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:09:39.676229 15821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:09:39.677778 15821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:09:39.677963 15821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:09:39.678133 15821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:09:39.683122 15827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:09:39.683231 15821 server_base.cc:1034] running on GCE node
W20260620 00:09:39.683142 15826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:39.683145 15829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:09:39.683543 15821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:09:39.683792 15821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:09:39.684938 15821 hybrid_clock.cc:648] HybridClock initialized: now 1781914179684928 us; error 28 us; skew 500 ppm
Jun 20 00:09:39 dist-test-slave-grqt krb5kdc[15714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:09:39.687642 15821 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:09:39.688174 15821 webserver.cc:460] Webserver started at http://127.0.98.1:44353/ using document root <none> and password file <none>
I20260620 00:09:39.688361 15821 fs_manager.cc:359] Metadata directory not provided
I20260620 00:09:39.688401 15821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:09:39.688488 15821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:09:39.689620 15821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/data/instance:
uuid: "50bf521ab90d423eb1f0b0c9661b8161"
format_stamp: "Formatted at 2026-06-20 00:09:39 on dist-test-slave-grqt"
server_key: "38a5349f361d02332db20f36b73990f3"
server_key_iv: "0cbc37e16844182871a9462422c0e888"
server_key_version: "encryptionkey@0"
I20260620 00:09:39.689980 15821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/wal/instance:
uuid: "50bf521ab90d423eb1f0b0c9661b8161"
format_stamp: "Formatted at 2026-06-20 00:09:39 on dist-test-slave-grqt"
server_key: "38a5349f361d02332db20f36b73990f3"
server_key_iv: "0cbc37e16844182871a9462422c0e888"
server_key_version: "encryptionkey@0"
I20260620 00:09:39.691324 15821 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:09:39.692071 15836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:39.692235 15821 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:09:39.692318 15821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/wal
uuid: "50bf521ab90d423eb1f0b0c9661b8161"
format_stamp: "Formatted at 2026-06-20 00:09:39 on dist-test-slave-grqt"
server_key: "38a5349f361d02332db20f36b73990f3"
server_key_iv: "0cbc37e16844182871a9462422c0e888"
server_key_version: "encryptionkey@0"
I20260620 00:09:39.692373 15821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:09:39.702443 15821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:09:39.704475 15821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:09:39.704595 15821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:09:39.704774 15821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:09:39.705015 15821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:09:39.705045 15821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:39.705066 15821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:09:39.705080 15821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:39.710911 15821 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39261
I20260620 00:09:39.710956 15949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39261 every 8 connection(s)
I20260620 00:09:39.711288 15821 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:09:39.717402 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 15821
I20260620 00:09:39.717514 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/wal/instance
Jun 20 00:09:39 dist-test-slave-grqt krb5kdc[15714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:09:39.717756 392 external_mini_cluster.cc:1428] Setting key 128f1eb51c3728190798251c9d13bad9
I20260620 00:09:39.721119 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:39.712026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41581 (local address 127.0.98.62:33689)
0620 00:09:39.712104 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:39.712107 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:39.712291 (+ 184us) server_negotiation.cc:407] Connection header received
0620 00:09:39.712825 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:39.712827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:39.712844 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:39.712897 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:09:39.713971 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:39.714488 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:39.715488 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:39.715714 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:39.715793 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:39.718232 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:09:39.718233 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:09:39.718233 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:09:39.718240 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:09:39.719729 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:39.720150 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:39.720151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:39.720152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:39.720211 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:39.720465 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:39.720466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:39.720466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:39.720581 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:09:39.720659 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:39.720931 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 00:09:39.721018 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 00:09:39.721304 15952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:39.712011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.1:41581)
0620 00:09:39.712167 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:39.712203 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:39.712716 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:39.712984 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:39.712988 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:39.713668 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:09:39.713867 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:39.713872 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:39.714591 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:39.714592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:39.715392 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:39.715394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:39.715799 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:39.715801 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:39.715927 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:39.716412 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:09:39.716421 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:09:39.718141 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:09:39.719831 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:39.719838 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:39.719841 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:39.720073 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:39.720279 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:39.720280 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:39.720281 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:39.720397 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:39.720650 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:09:39.720657 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:09:39.720799 (+ 142us) client_negotiation.cc:769] Sending connection context
0620 00:09:39.720911 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 00:09:39.721070 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":76,"threads_started":1}
I20260620 00:09:39.721851 15950 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33689
I20260620 00:09:39.721966 15950 heartbeater.cc:464] Registering TS with master...
I20260620 00:09:39.722177 15950 heartbeater.cc:511] Master 127.0.98.62:33689 requested a full tablet report, sending...
I20260620 00:09:39.722666 15759 ts_manager.cc:194] Registered new tserver with Master: 50bf521ab90d423eb1f0b0c9661b8161 (127.0.98.1:39261)
I20260620 00:09:39.723294 15759 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41581
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:09:39.743805 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33689
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:09:39.827318 15955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:09:39.827595 15955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:09:39.827638 15955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:09:39.828840 15955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:09:39.828914 15955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:09:39.828984 15955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:09:39.830559 15955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:09:39.830855 15955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:09:39.831107 15955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:09:39.836575 15961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:39.836606 15963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:39.836622 15960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:09:39.836736 15955 server_base.cc:1034] running on GCE node
I20260620 00:09:39.837065 15955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:09:39.837422 15955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:09:39.838605 15955 hybrid_clock.cc:648] HybridClock initialized: now 1781914179838586 us; error 36 us; skew 500 ppm
Jun 20 00:09:39 dist-test-slave-grqt krb5kdc[15714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:09:39.841212 15955 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:09:39.841769 15955 webserver.cc:460] Webserver started at http://127.0.98.2:45341/ using document root <none> and password file <none>
I20260620 00:09:39.842024 15955 fs_manager.cc:359] Metadata directory not provided
I20260620 00:09:39.842098 15955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:09:39.842247 15955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:09:39.843335 15955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/data/instance:
uuid: "cbce6b11c84e4a0faefed80f8db30eaa"
format_stamp: "Formatted at 2026-06-20 00:09:39 on dist-test-slave-grqt"
server_key: "6ef693e56ec2dc7e4b3e09dc550ba60f"
server_key_iv: "28a0b898c975def66c3bc3a00bc0a42f"
server_key_version: "encryptionkey@0"
I20260620 00:09:39.843719 15955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cbce6b11c84e4a0faefed80f8db30eaa"
format_stamp: "Formatted at 2026-06-20 00:09:39 on dist-test-slave-grqt"
server_key: "6ef693e56ec2dc7e4b3e09dc550ba60f"
server_key_iv: "28a0b898c975def66c3bc3a00bc0a42f"
server_key_version: "encryptionkey@0"
I20260620 00:09:39.845148 15955 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:09:39.846035 15970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:39.846212 15955 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:09:39.846319 15955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/wal
uuid: "cbce6b11c84e4a0faefed80f8db30eaa"
format_stamp: "Formatted at 2026-06-20 00:09:39 on dist-test-slave-grqt"
server_key: "6ef693e56ec2dc7e4b3e09dc550ba60f"
server_key_iv: "28a0b898c975def66c3bc3a00bc0a42f"
server_key_version: "encryptionkey@0"
I20260620 00:09:39.846397 15955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:09:39.865227 15955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:09:39.867806 15955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:09:39.867983 15955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:09:39.868233 15955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:09:39.868556 15955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:09:39.868618 15955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:39.868685 15955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:09:39.868716 15955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:39.874749 15955 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46097
I20260620 00:09:39.874791 16083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46097 every 8 connection(s)
I20260620 00:09:39.875120 15955 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:09:39.879134 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 15955
I20260620 00:09:39.879223 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:09:39.879467 392 external_mini_cluster.cc:1428] Setting key 44dcb9cf44e8f654611423f67f218c25
Jun 20 00:09:39 dist-test-slave-grqt krb5kdc[15714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:09:39.885620 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:39.875813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56843 (local address 127.0.98.62:33689)
0620 00:09:39.875889 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:39.875891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:39.876121 (+ 230us) server_negotiation.cc:407] Connection header received
0620 00:09:39.876669 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:39.876671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:39.876682 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:39.876742 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:09:39.877843 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:39.878351 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:39.879497 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:39.879740 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:39.879816 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:39.882299 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:09:39.882301 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:09:39.882302 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:09:39.882313 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:09:39.883925 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:39.884475 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:39.884476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:39.884476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:39.884532 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:39.884866 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:39.884867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:39.884867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:39.884975 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:09:39.885025 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:39.885438 (+ 413us) server_negotiation.cc:299] Negotiation successful
0620 00:09:39.885525 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 00:09:39.885929 16086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:39.875819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.2:56843)
0620 00:09:39.876005 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:39.876050 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:39.876525 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:39.876828 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:39.876831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:39.877525 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:09:39.877733 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:39.877739 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:39.878469 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:39.878470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:39.879404 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:39.879406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:39.879827 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:39.879828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:39.879971 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:39.880474 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:09:39.880483 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:09:39.882197 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:09:39.884035 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:39.884044 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:39.884047 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:39.884399 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:39.884633 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:39.884634 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:39.884634 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:39.884794 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:39.885045 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:09:39.885052 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:09:39.885246 (+ 194us) client_negotiation.cc:769] Sending connection context
0620 00:09:39.885410 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 00:09:39.885633 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":98,"threads_started":1}
I20260620 00:09:39.886487 16084 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33689
I20260620 00:09:39.886591 16084 heartbeater.cc:464] Registering TS with master...
I20260620 00:09:39.886796 16084 heartbeater.cc:511] Master 127.0.98.62:33689 requested a full tablet report, sending...
I20260620 00:09:39.887187 15759 ts_manager.cc:194] Registered new tserver with Master: cbce6b11c84e4a0faefed80f8db30eaa (127.0.98.2:46097)
I20260620 00:09:39.887565 15759 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56843
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:09:39.906147 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33689
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:09:39.986590 16089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:09:39.986802 16089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:09:39.986827 16089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:09:39.987916 16089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:09:39.987951 16089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:09:39.987977 16089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:09:39.989521 16089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:09:39.989732 16089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:09:39.989909 16089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:09:39.995508 16094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:09:39.995581 16089 server_base.cc:1034] running on GCE node
W20260620 00:09:39.995451 16097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:09:39.995451 16095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:09:39.995959 16089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:09:39.996254 16089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:09:39.997473 16089 hybrid_clock.cc:648] HybridClock initialized: now 1781914179997457 us; error 31 us; skew 500 ppm
Jun 20 00:09:39 dist-test-slave-grqt krb5kdc[15714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:09:40.000011 16089 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:09:40.000514 16089 webserver.cc:460] Webserver started at http://127.0.98.3:33931/ using document root <none> and password file <none>
I20260620 00:09:40.000716 16089 fs_manager.cc:359] Metadata directory not provided
I20260620 00:09:40.000757 16089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:09:40.000855 16089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:09:40.002060 16089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/data/instance:
uuid: "49efdd3061be4caeb77db9887e9456f6"
format_stamp: "Formatted at 2026-06-20 00:09:39 on dist-test-slave-grqt"
server_key: "589538078972f151c4c06ec4b247b957"
server_key_iv: "2fafc5451917ca96272979c9703da4f2"
server_key_version: "encryptionkey@0"
I20260620 00:09:40.002385 16089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/wal/instance:
uuid: "49efdd3061be4caeb77db9887e9456f6"
format_stamp: "Formatted at 2026-06-20 00:09:39 on dist-test-slave-grqt"
server_key: "589538078972f151c4c06ec4b247b957"
server_key_iv: "2fafc5451917ca96272979c9703da4f2"
server_key_version: "encryptionkey@0"
I20260620 00:09:40.003827 16089 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:09:40.004822 16104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:40.005038 16089 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:09:40.005131 16089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/wal
uuid: "49efdd3061be4caeb77db9887e9456f6"
format_stamp: "Formatted at 2026-06-20 00:09:39 on dist-test-slave-grqt"
server_key: "589538078972f151c4c06ec4b247b957"
server_key_iv: "2fafc5451917ca96272979c9703da4f2"
server_key_version: "encryptionkey@0"
I20260620 00:09:40.005192 16089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:09:40.018282 16089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:09:40.020702 16089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:09:40.020823 16089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:09:40.021008 16089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:09:40.021317 16089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:09:40.021353 16089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:40.021433 16089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:09:40.021467 16089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:09:40.027259 16089 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40275
I20260620 00:09:40.027303 16217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40275 every 8 connection(s)
I20260620 00:09:40.027616 16089 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:09:40.030864 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 16089
I20260620 00:09:40.030969 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:09:40.031173 392 external_mini_cluster.cc:1428] Setting key 72bf122da358db7beeea44ee986d937d
Jun 20 00:09:40 dist-test-slave-grqt krb5kdc[15714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:09:40.037312 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:40.028271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45481 (local address 127.0.98.62:33689)
0620 00:09:40.028314 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:40.028316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:40.028524 (+ 208us) server_negotiation.cc:407] Connection header received
0620 00:09:40.028982 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:40.028983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:40.028993 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:40.029047 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:09:40.030015 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:40.030516 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:40.031894 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:40.032190 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:40.032247 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:40.034577 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:09:40.034578 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:09:40.034579 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:09:40.034586 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:09:40.035912 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:40.036333 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:40.036334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:40.036335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:40.036392 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:40.036653 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:40.036654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:40.036655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:40.036777 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:09:40.036855 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:40.037130 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 00:09:40.037216 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 00:09:40.037521 16220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:40.028259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.3:45481)
0620 00:09:40.028393 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:40.028429 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:40.028892 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:40.029103 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:40.029106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:40.029722 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:09:40.029924 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:40.029929 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:40.030644 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:40.030670 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:40.031820 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:40.031822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:40.032267 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:40.032267 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:40.032370 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:40.032990 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:09:40.032998 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:09:40.034470 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:09:40.036020 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:40.036027 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:40.036029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:40.036257 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:40.036470 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:09:40.036471 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:09:40.036471 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:09:40.036581 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:09:40.036844 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:09:40.036850 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:09:40.036995 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 00:09:40.037101 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 00:09:40.037261 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":71,"threads_started":1}
I20260620 00:09:40.037997 16218 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33689
I20260620 00:09:40.038075 16218 heartbeater.cc:464] Registering TS with master...
I20260620 00:09:40.038281 16218 heartbeater.cc:511] Master 127.0.98.62:33689 requested a full tablet report, sending...
I20260620 00:09:40.038650 15759 ts_manager.cc:194] Registered new tserver with Master: 49efdd3061be4caeb77db9887e9456f6 (127.0.98.3:40275)
I20260620 00:09:40.039036 15759 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45481
I20260620 00:09:40.042795 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:09:40.051013 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:40.044149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44380 (local address 127.0.98.62:33689)
0620 00:09:40.044246 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:40.044248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:40.044544 (+ 296us) server_negotiation.cc:407] Connection header received
0620 00:09:40.044598 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:40.044599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:40.044609 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:40.044671 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:09:40.045226 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:40.045798 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:40.046560 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:40.046743 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:40.046814 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:40.047931 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:09:40.047933 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:09:40.047933 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:09:40.047940 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:09:40.049712 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:40.050165 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:40.050166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:40.050167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:40.050232 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:09:40.050430 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:09:40.050431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:09:40.050432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:09:40.050591 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:09:40.050698 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:40.050811 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 00:09:40.050903 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 00:09:40.052690 15759 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44380:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:09:40.054352 15759 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:09:40.060672 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:40.056467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48194 (local address 127.0.98.2:46097)
0620 00:09:40.056737 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:40.056741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:40.056778 (+ 37us) server_negotiation.cc:407] Connection header received
0620 00:09:40.056870 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:40.056873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:40.056895 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:40.057004 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:09:40.057158 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:40.058518 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:40.059740 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:40.060232 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:40.060329 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:40.060396 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:40.060527 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 00:09:40.060572 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":83,"threads_started":1}
I20260620 00:09:40.060721 16226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:40.056354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40275 (local address 127.0.0.1:40782)
0620 00:09:40.056474 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:40.056516 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:40.056579 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:40.056960 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:40.056964 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:40.056967 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:09:40.057039 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:40.057041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:40.058482 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:40.058483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:40.059420 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:40.059423 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:40.060400 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:40.060401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:40.060477 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:40.060484 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 00:09:40.060516 (+ 32us) client_negotiation.cc:240] Negotiation successful
0620 00:09:40.060564 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":38,"thread_start_us":74,"threads_started":1}
I20260620 00:09:40.060820 16229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:40.056538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40782 (local address 127.0.98.3:40275)
0620 00:09:40.056692 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:40.056698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:40.056710 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:09:40.056754 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:40.056757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:40.056772 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:40.056863 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:09:40.057152 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:40.058359 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:40.059521 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:40.060280 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:40.060611 (+ 331us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:40.060673 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:40.060699 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 00:09:40.060743 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":83,"threads_started":1}
I20260620 00:09:40.060909 16231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:40.056778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59326 (local address 127.0.98.1:39261)
0620 00:09:40.056936 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:40.056940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:40.057080 (+ 140us) server_negotiation.cc:407] Connection header received
0620 00:09:40.057147 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:40.057150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:40.057165 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:40.057235 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:09:40.057461 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:40.058204 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:40.059652 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:40.060334 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:40.060397 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:40.060454 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:40.060758 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 00:09:40.060801 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
I20260620 00:09:40.060724 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:40.056567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39261 (local address 127.0.0.1:59326)
0620 00:09:40.056950 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:40.056959 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:40.057021 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:40.057307 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:40.057308 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:40.057309 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:09:40.057337 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:40.057358 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:40.058314 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:40.058315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:40.059560 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:40.059562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:40.060485 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:40.060486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:40.060541 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:40.060545 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:09:40.060589 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 00:09:40.060624 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":61,"threads_started":1}
I20260620 00:09:40.061198 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:40.056358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46097 (local address 127.0.0.1:48194)
0620 00:09:40.056724 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:40.056735 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:40.056787 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:40.057023 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:40.057024 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:40.057026 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:09:40.057081 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:40.057083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:40.058611 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:40.058612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:40.059678 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:40.059680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:40.060330 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:40.060331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:40.060413 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:40.060426 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 00:09:40.060500 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 00:09:40.060558 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":67,"threads_started":1}
I20260620 00:09:40.061818 15884 tablet_service.cc:1467] Processing CreateTablet for tablet 7381663ac53b484c9de17dee81b1bb93 (DEFAULT_TABLE table=test-table [id=977995cb4a5e4bc1925cb56f759601fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:09:40.061818 16152 tablet_service.cc:1467] Processing CreateTablet for tablet 7381663ac53b484c9de17dee81b1bb93 (DEFAULT_TABLE table=test-table [id=977995cb4a5e4bc1925cb56f759601fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:09:40.062024 16018 tablet_service.cc:1467] Processing CreateTablet for tablet 7381663ac53b484c9de17dee81b1bb93 (DEFAULT_TABLE table=test-table [id=977995cb4a5e4bc1925cb56f759601fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:09:40.062173 16152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7381663ac53b484c9de17dee81b1bb93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:09:40.062173 15884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7381663ac53b484c9de17dee81b1bb93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:09:40.062274 16018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7381663ac53b484c9de17dee81b1bb93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:09:40.064867 16232 tablet_bootstrap.cc:492] T 7381663ac53b484c9de17dee81b1bb93 P cbce6b11c84e4a0faefed80f8db30eaa: Bootstrap starting.
I20260620 00:09:40.065338 16233 tablet_bootstrap.cc:492] T 7381663ac53b484c9de17dee81b1bb93 P 50bf521ab90d423eb1f0b0c9661b8161: Bootstrap starting.
I20260620 00:09:40.065519 16232 tablet_bootstrap.cc:654] T 7381663ac53b484c9de17dee81b1bb93 P cbce6b11c84e4a0faefed80f8db30eaa: Neither blocks nor log segments found. Creating new log.
I20260620 00:09:40.065843 16234 tablet_bootstrap.cc:492] T 7381663ac53b484c9de17dee81b1bb93 P 49efdd3061be4caeb77db9887e9456f6: Bootstrap starting.
I20260620 00:09:40.066108 16233 tablet_bootstrap.cc:654] T 7381663ac53b484c9de17dee81b1bb93 P 50bf521ab90d423eb1f0b0c9661b8161: Neither blocks nor log segments found. Creating new log.
I20260620 00:09:40.066124 16232 log.cc:824] T 7381663ac53b484c9de17dee81b1bb93 P cbce6b11c84e4a0faefed80f8db30eaa: Log is configured to *not* fsync() on all Append() calls
I20260620 00:09:40.066481 16234 tablet_bootstrap.cc:654] T 7381663ac53b484c9de17dee81b1bb93 P 49efdd3061be4caeb77db9887e9456f6: Neither blocks nor log segments found. Creating new log.
I20260620 00:09:40.066686 16233 log.cc:824] T 7381663ac53b484c9de17dee81b1bb93 P 50bf521ab90d423eb1f0b0c9661b8161: Log is configured to *not* fsync() on all Append() calls
I20260620 00:09:40.066941 16232 tablet_bootstrap.cc:492] T 7381663ac53b484c9de17dee81b1bb93 P cbce6b11c84e4a0faefed80f8db30eaa: No bootstrap required, opened a new log
I20260620 00:09:40.067031 16232 ts_tablet_manager.cc:1397] T 7381663ac53b484c9de17dee81b1bb93 P cbce6b11c84e4a0faefed80f8db30eaa: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:09:40.067018 16234 log.cc:824] T 7381663ac53b484c9de17dee81b1bb93 P 49efdd3061be4caeb77db9887e9456f6: Log is configured to *not* fsync() on all Append() calls
I20260620 00:09:40.067435 16233 tablet_bootstrap.cc:492] T 7381663ac53b484c9de17dee81b1bb93 P 50bf521ab90d423eb1f0b0c9661b8161: No bootstrap required, opened a new log
I20260620 00:09:40.067507 16233 ts_tablet_manager.cc:1397] T 7381663ac53b484c9de17dee81b1bb93 P 50bf521ab90d423eb1f0b0c9661b8161: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:09:40.067700 16234 tablet_bootstrap.cc:492] T 7381663ac53b484c9de17dee81b1bb93 P 49efdd3061be4caeb77db9887e9456f6: No bootstrap required, opened a new log
I20260620 00:09:40.067781 16234 ts_tablet_manager.cc:1397] T 7381663ac53b484c9de17dee81b1bb93 P 49efdd3061be4caeb77db9887e9456f6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:09:40.069001 16232 raft_consensus.cc:348] T 7381663ac53b484c9de17dee81b1bb93 P cbce6b11c84e4a0faefed80f8db30eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49efdd3061be4caeb77db9887e9456f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40275 } } peers { permanent_uuid: "cbce6b11c84e4a0faefed80f8db30eaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46097 } } peers { permanent_uuid: "50bf521ab90d423eb1f0b0c9661b8161" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39261 } }
I20260620 00:09:40.069207 16232 raft_consensus.cc:374] T 7381663ac53b484c9de17dee81b1bb93 P cbce6b11c84e4a0faefed80f8db30eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:09:40.069244 16232 raft_consensus.cc:729] T 7381663ac53b484c9de17dee81b1bb93 P cbce6b11c84e4a0faefed80f8db30eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbce6b11c84e4a0faefed80f8db30eaa, State: Initialized, Role: FOLLOWER
I20260620 00:09:40.069424 16232 consensus_queue.cc:260] T 7381663ac53b484c9de17dee81b1bb93 P cbce6b11c84e4a0faefed80f8db30eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49efdd3061be4caeb77db9887e9456f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40275 } } peers { permanent_uuid: "cbce6b11c84e4a0faefed80f8db30eaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46097 } } peers { permanent_uuid: "50bf521ab90d423eb1f0b0c9661b8161" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39261 } }
I20260620 00:09:40.069439 16233 raft_consensus.cc:348] T 7381663ac53b484c9de17dee81b1bb93 P 50bf521ab90d423eb1f0b0c9661b8161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49efdd3061be4caeb77db9887e9456f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40275 } } peers { permanent_uuid: "cbce6b11c84e4a0faefed80f8db30eaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46097 } } peers { permanent_uuid: "50bf521ab90d423eb1f0b0c9661b8161" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39261 } }
I20260620 00:09:40.069607 16233 raft_consensus.cc:374] T 7381663ac53b484c9de17dee81b1bb93 P 50bf521ab90d423eb1f0b0c9661b8161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:09:40.069356 16234 raft_consensus.cc:348] T 7381663ac53b484c9de17dee81b1bb93 P 49efdd3061be4caeb77db9887e9456f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49efdd3061be4caeb77db9887e9456f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40275 } } peers { permanent_uuid: "cbce6b11c84e4a0faefed80f8db30eaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46097 } } peers { permanent_uuid: "50bf521ab90d423eb1f0b0c9661b8161" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39261 } }
I20260620 00:09:40.069650 16233 raft_consensus.cc:729] T 7381663ac53b484c9de17dee81b1bb93 P 50bf521ab90d423eb1f0b0c9661b8161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50bf521ab90d423eb1f0b0c9661b8161, State: Initialized, Role: FOLLOWER
I20260620 00:09:40.069664 16234 raft_consensus.cc:374] T 7381663ac53b484c9de17dee81b1bb93 P 49efdd3061be4caeb77db9887e9456f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:09:40.069725 16232 ts_tablet_manager.cc:1428] T 7381663ac53b484c9de17dee81b1bb93 P cbce6b11c84e4a0faefed80f8db30eaa: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:09:40.069720 16234 raft_consensus.cc:729] T 7381663ac53b484c9de17dee81b1bb93 P 49efdd3061be4caeb77db9887e9456f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49efdd3061be4caeb77db9887e9456f6, State: Initialized, Role: FOLLOWER
I20260620 00:09:40.069765 16233 consensus_queue.cc:260] T 7381663ac53b484c9de17dee81b1bb93 P 50bf521ab90d423eb1f0b0c9661b8161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49efdd3061be4caeb77db9887e9456f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40275 } } peers { permanent_uuid: "cbce6b11c84e4a0faefed80f8db30eaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46097 } } peers { permanent_uuid: "50bf521ab90d423eb1f0b0c9661b8161" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39261 } }
I20260620 00:09:40.069854 16084 heartbeater.cc:503] Master 127.0.98.62:33689 was elected leader, sending a full tablet report...
I20260620 00:09:40.070027 16233 ts_tablet_manager.cc:1428] T 7381663ac53b484c9de17dee81b1bb93 P 50bf521ab90d423eb1f0b0c9661b8161: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 00:09:40.070225 15950 heartbeater.cc:503] Master 127.0.98.62:33689 was elected leader, sending a full tablet report...
I20260620 00:09:40.070179 16234 consensus_queue.cc:260] T 7381663ac53b484c9de17dee81b1bb93 P 49efdd3061be4caeb77db9887e9456f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49efdd3061be4caeb77db9887e9456f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40275 } } peers { permanent_uuid: "cbce6b11c84e4a0faefed80f8db30eaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46097 } } peers { permanent_uuid: "50bf521ab90d423eb1f0b0c9661b8161" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39261 } }
I20260620 00:09:40.070452 16234 ts_tablet_manager.cc:1428] T 7381663ac53b484c9de17dee81b1bb93 P 49efdd3061be4caeb77db9887e9456f6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:09:40.070480 16218 heartbeater.cc:503] Master 127.0.98.62:33689 was elected leader, sending a full tablet report...
W20260620 00:09:40.125686 16085 tablet.cc:2369] T 7381663ac53b484c9de17dee81b1bb93 P cbce6b11c84e4a0faefed80f8db30eaa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 00:09:40.211964 15951 tablet.cc:2369] T 7381663ac53b484c9de17dee81b1bb93 P 50bf521ab90d423eb1f0b0c9661b8161: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:09:40.234863 16239 raft_consensus.cc:482] T 7381663ac53b484c9de17dee81b1bb93 P 50bf521ab90d423eb1f0b0c9661b8161 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:09:40.234982 16239 raft_consensus.cc:504] T 7381663ac53b484c9de17dee81b1bb93 P 50bf521ab90d423eb1f0b0c9661b8161 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49efdd3061be4caeb77db9887e9456f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40275 } } peers { permanent_uuid: "cbce6b11c84e4a0faefed80f8db30eaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46097 } } peers { permanent_uuid: "50bf521ab90d423eb1f0b0c9661b8161" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39261 } }
I20260620 00:09:40.235361 16239 leader_election.cc:288] T 7381663ac53b484c9de17dee81b1bb93 P 50bf521ab90d423eb1f0b0c9661b8161 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49efdd3061be4caeb77db9887e9456f6 (127.0.98.3:40275), cbce6b11c84e4a0faefed80f8db30eaa (127.0.98.2:46097)
I20260620 00:09:40.238348 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:40.235722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43347 (local address 127.0.98.2:46097)
0620 00:09:40.235772 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:40.235774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:40.235804 (+ 30us) server_negotiation.cc:407] Connection header received
0620 00:09:40.235888 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:40.235889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:40.235901 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:40.235961 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:09:40.236155 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:40.236606 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:40.237605 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:40.237998 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:40.238052 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:40.238061 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:40.238226 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 00:09:40.238275 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 00:09:40.238384 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:40.235586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46097 (local address 127.0.98.1:43347)
0620 00:09:40.235740 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:40.235751 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:40.235817 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:40.235974 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:40.235975 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:40.235976 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:09:40.236082 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:40.236084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:40.236713 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:40.236714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:40.237524 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:40.237526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:40.238078 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:40.238079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:40.238149 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:40.238153 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:09:40.238206 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 00:09:40.238237 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":14,"thread_start_us":74,"threads_started":1}
I20260620 00:09:40.238413 16241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:40.235536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40275 (local address 127.0.98.1:59929)
0620 00:09:40.235661 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:40.235674 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:09:40.235734 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:09:40.235900 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:09:40.235901 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:09:40.235904 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:09:40.235982 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:40.235985 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:40.236679 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:40.236680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:40.237581 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:09:40.237585 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:09:40.238146 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:09:40.238147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:09:40.238220 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:40.238224 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:09:40.238277 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 00:09:40.238314 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260620 00:09:40.238418 16229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:40.235700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59929 (local address 127.0.98.3:40275)
0620 00:09:40.235746 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:40.235749 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:40.235761 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:09:40.235795 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:40.235797 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:40.235808 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:40.235877 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:09:40.236064 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:40.236582 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:40.237684 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:40.238048 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:40.238121 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:40.238130 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:40.238301 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 00:09:40.238345 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 00:09:40.238718 16038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7381663ac53b484c9de17dee81b1bb93" candidate_uuid: "50bf521ab90d423eb1f0b0c9661b8161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbce6b11c84e4a0faefed80f8db30eaa" is_pre_election: true
I20260620 00:09:40.238729 16172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7381663ac53b484c9de17dee81b1bb93" candidate_uuid: "50bf521ab90d423eb1f0b0c9661b8161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49efdd3061be4caeb77db9887e9456f6" is_pre_election: true
I20260620 00:09:40.238857 16038 raft_consensus.cc:2454] T 7381663ac53b484c9de17dee81b1bb93 P cbce6b11c84e4a0faefed80f8db30eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50bf521ab90d423eb1f0b0c9661b8161 in term 0.
I20260620 00:09:40.238895 16172 raft_consensus.cc:2454] T 7381663ac53b484c9de17dee81b1bb93 P 49efdd3061be4caeb77db9887e9456f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50bf521ab90d423eb1f0b0c9661b8161 in term 0.
I20260620 00:09:40.239087 15837 leader_election.cc:302] T 7381663ac53b484c9de17dee81b1bb93 P 50bf521ab90d423eb1f0b0c9661b8161 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50bf521ab90d423eb1f0b0c9661b8161, cbce6b11c84e4a0faefed80f8db30eaa; no voters:
I20260620 00:09:40.239219 16239 raft_consensus.cc:2781] T 7381663ac53b484c9de17dee81b1bb93 P 50bf521ab90d423eb1f0b0c9661b8161 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:09:40.239272 16239 raft_consensus.cc:482] T 7381663ac53b484c9de17dee81b1bb93 P 50bf521ab90d423eb1f0b0c9661b8161 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:09:40.239288 16239 raft_consensus.cc:3037] T 7381663ac53b484c9de17dee81b1bb93 P 50bf521ab90d423eb1f0b0c9661b8161 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:09:40.240010 16239 raft_consensus.cc:504] T 7381663ac53b484c9de17dee81b1bb93 P 50bf521ab90d423eb1f0b0c9661b8161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49efdd3061be4caeb77db9887e9456f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40275 } } peers { permanent_uuid: "cbce6b11c84e4a0faefed80f8db30eaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46097 } } peers { permanent_uuid: "50bf521ab90d423eb1f0b0c9661b8161" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39261 } }
I20260620 00:09:40.240159 16239 leader_election.cc:288] T 7381663ac53b484c9de17dee81b1bb93 P 50bf521ab90d423eb1f0b0c9661b8161 [CANDIDATE]: Term 1 election: Requested vote from peers 49efdd3061be4caeb77db9887e9456f6 (127.0.98.3:40275), cbce6b11c84e4a0faefed80f8db30eaa (127.0.98.2:46097)
I20260620 00:09:40.240319 16172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7381663ac53b484c9de17dee81b1bb93" candidate_uuid: "50bf521ab90d423eb1f0b0c9661b8161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49efdd3061be4caeb77db9887e9456f6"
I20260620 00:09:40.240345 16038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7381663ac53b484c9de17dee81b1bb93" candidate_uuid: "50bf521ab90d423eb1f0b0c9661b8161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbce6b11c84e4a0faefed80f8db30eaa"
I20260620 00:09:40.240403 16172 raft_consensus.cc:3037] T 7381663ac53b484c9de17dee81b1bb93 P 49efdd3061be4caeb77db9887e9456f6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:09:40.240413 16038 raft_consensus.cc:3037] T 7381663ac53b484c9de17dee81b1bb93 P cbce6b11c84e4a0faefed80f8db30eaa [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:09:40.241251 16172 raft_consensus.cc:2454] T 7381663ac53b484c9de17dee81b1bb93 P 49efdd3061be4caeb77db9887e9456f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50bf521ab90d423eb1f0b0c9661b8161 in term 1.
I20260620 00:09:40.241251 16038 raft_consensus.cc:2454] T 7381663ac53b484c9de17dee81b1bb93 P cbce6b11c84e4a0faefed80f8db30eaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50bf521ab90d423eb1f0b0c9661b8161 in term 1.
I20260620 00:09:40.241465 15837 leader_election.cc:302] T 7381663ac53b484c9de17dee81b1bb93 P 50bf521ab90d423eb1f0b0c9661b8161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50bf521ab90d423eb1f0b0c9661b8161, cbce6b11c84e4a0faefed80f8db30eaa; no voters:
I20260620 00:09:40.241657 16239 raft_consensus.cc:2781] T 7381663ac53b484c9de17dee81b1bb93 P 50bf521ab90d423eb1f0b0c9661b8161 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:09:40.241822 16239 raft_consensus.cc:686] T 7381663ac53b484c9de17dee81b1bb93 P 50bf521ab90d423eb1f0b0c9661b8161 [term 1 LEADER]: Becoming Leader. State: Replica: 50bf521ab90d423eb1f0b0c9661b8161, State: Running, Role: LEADER
I20260620 00:09:40.241920 16239 consensus_queue.cc:237] T 7381663ac53b484c9de17dee81b1bb93 P 50bf521ab90d423eb1f0b0c9661b8161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49efdd3061be4caeb77db9887e9456f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40275 } } peers { permanent_uuid: "cbce6b11c84e4a0faefed80f8db30eaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46097 } } peers { permanent_uuid: "50bf521ab90d423eb1f0b0c9661b8161" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39261 } }
I20260620 00:09:40.242727 15758 catalog_manager.cc:5495] T 7381663ac53b484c9de17dee81b1bb93 P 50bf521ab90d423eb1f0b0c9661b8161 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50bf521ab90d423eb1f0b0c9661b8161 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "50bf521ab90d423eb1f0b0c9661b8161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49efdd3061be4caeb77db9887e9456f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbce6b11c84e4a0faefed80f8db30eaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50bf521ab90d423eb1f0b0c9661b8161" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39261 } health_report { overall_health: HEALTHY } } }
W20260620 00:09:40.278155 16219 tablet.cc:2369] T 7381663ac53b484c9de17dee81b1bb93 P 49efdd3061be4caeb77db9887e9456f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:09:40.291320 16231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:40.289007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59340 (local address 127.0.98.1:39261)
0620 00:09:40.289065 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 00:09:40.289066 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:09:40.289075 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:09:40.289157 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:09:40.289157 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:09:40.289178 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:09:40.289239 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:09:40.289355 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:40.289878 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:40.290618 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:09:40.290801 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:09:40.290857 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:09:40.290985 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:09:40.291086 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:09:40.291155 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 00:09:40.291197 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 00:09:40.291213 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 00:09:40.293493 16172 raft_consensus.cc:1261] T 7381663ac53b484c9de17dee81b1bb93 P 49efdd3061be4caeb77db9887e9456f6 [term 1 FOLLOWER]: Refusing update from remote peer 50bf521ab90d423eb1f0b0c9661b8161: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:09:40.293514 16038 raft_consensus.cc:1261] T 7381663ac53b484c9de17dee81b1bb93 P cbce6b11c84e4a0faefed80f8db30eaa [term 1 FOLLOWER]: Refusing update from remote peer 50bf521ab90d423eb1f0b0c9661b8161: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:09:40.293892 16239 consensus_queue.cc:1035] T 7381663ac53b484c9de17dee81b1bb93 P 50bf521ab90d423eb1f0b0c9661b8161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49efdd3061be4caeb77db9887e9456f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:09:40.294040 16243 consensus_queue.cc:1035] T 7381663ac53b484c9de17dee81b1bb93 P 50bf521ab90d423eb1f0b0c9661b8161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbce6b11c84e4a0faefed80f8db30eaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:09:40.297017 16246 mvcc.cc:205] Tried to move back new op lower bound from 7298720482478698496 to 7298720482271612928. Current Snapshot: MvccSnapshot[applied={T|T < 7298720482478698496 or (T in {7298720482478698496})}]
I20260620 00:09:40.298115 16247 mvcc.cc:205] Tried to move back new op lower bound from 7298720482478698496 to 7298720482271612928. Current Snapshot: MvccSnapshot[applied={T|T < 7298720482478698496 or (T in {7298720482478698496})}]
I20260620 00:09:40.299052 15759 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:09:40.299160 15759 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:09:40.299947 15759 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0: Sending DeleteTablet for 3 replicas of tablet 7381663ac53b484c9de17dee81b1bb93
I20260620 00:09:40.300379 16018 tablet_service.cc:1514] Processing DeleteTablet for tablet 7381663ac53b484c9de17dee81b1bb93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:09:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48194
I20260620 00:09:40.300360 15884 tablet_service.cc:1514] Processing DeleteTablet for tablet 7381663ac53b484c9de17dee81b1bb93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:09:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59326
I20260620 00:09:40.300632 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 15821
I20260620 00:09:40.300659 16152 tablet_service.cc:1514] Processing DeleteTablet for tablet 7381663ac53b484c9de17dee81b1bb93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:09:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40782
I20260620 00:09:40.302083 16255 tablet_replica.cc:330] T 7381663ac53b484c9de17dee81b1bb93 P cbce6b11c84e4a0faefed80f8db30eaa: stopping tablet replica
I20260620 00:09:40.302237 16255 raft_consensus.cc:2229] T 7381663ac53b484c9de17dee81b1bb93 P cbce6b11c84e4a0faefed80f8db30eaa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:09:40.302382 16255 raft_consensus.cc:2258] T 7381663ac53b484c9de17dee81b1bb93 P cbce6b11c84e4a0faefed80f8db30eaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:09:40.302402 16257 tablet_replica.cc:330] T 7381663ac53b484c9de17dee81b1bb93 P 49efdd3061be4caeb77db9887e9456f6: stopping tablet replica
I20260620 00:09:40.302523 16257 raft_consensus.cc:2229] T 7381663ac53b484c9de17dee81b1bb93 P 49efdd3061be4caeb77db9887e9456f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:09:40.302717 16257 raft_consensus.cc:2258] T 7381663ac53b484c9de17dee81b1bb93 P 49efdd3061be4caeb77db9887e9456f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:09:40.302943 16255 ts_tablet_manager.cc:1905] T 7381663ac53b484c9de17dee81b1bb93 P cbce6b11c84e4a0faefed80f8db30eaa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:09:40.303156 16257 ts_tablet_manager.cc:1905] T 7381663ac53b484c9de17dee81b1bb93 P 49efdd3061be4caeb77db9887e9456f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:09:40.304822 16257 ts_tablet_manager.cc:1918] T 7381663ac53b484c9de17dee81b1bb93 P 49efdd3061be4caeb77db9887e9456f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:09:40.304929 16257 log.cc:1196] T 7381663ac53b484c9de17dee81b1bb93 P 49efdd3061be4caeb77db9887e9456f6: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/wal/wals/7381663ac53b484c9de17dee81b1bb93
I20260620 00:09:40.304997 16255 ts_tablet_manager.cc:1918] T 7381663ac53b484c9de17dee81b1bb93 P cbce6b11c84e4a0faefed80f8db30eaa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:09:40.305078 16255 log.cc:1196] T 7381663ac53b484c9de17dee81b1bb93 P cbce6b11c84e4a0faefed80f8db30eaa: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/wal/wals/7381663ac53b484c9de17dee81b1bb93
I20260620 00:09:40.305248 16257 ts_tablet_manager.cc:1939] T 7381663ac53b484c9de17dee81b1bb93 P 49efdd3061be4caeb77db9887e9456f6: Deleting consensus metadata
I20260620 00:09:40.305446 16255 ts_tablet_manager.cc:1939] T 7381663ac53b484c9de17dee81b1bb93 P cbce6b11c84e4a0faefed80f8db30eaa: Deleting consensus metadata
I20260620 00:09:40.305787 15746 catalog_manager.cc:4841] TS 49efdd3061be4caeb77db9887e9456f6 (127.0.98.3:40275): tablet 7381663ac53b484c9de17dee81b1bb93 (table test-table [id=977995cb4a5e4bc1925cb56f759601fe]) successfully deleted
I20260620 00:09:40.305889 15743 catalog_manager.cc:4841] TS cbce6b11c84e4a0faefed80f8db30eaa (127.0.98.2:46097): tablet 7381663ac53b484c9de17dee81b1bb93 (table test-table [id=977995cb4a5e4bc1925cb56f759601fe]) successfully deleted
W20260620 00:09:40.306887 15746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39261 (error 108)
I20260620 00:09:40.307130 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 15955
I20260620 00:09:40.307761 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:09:40.307245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39261 (local address 127.0.0.1:59354)
0620 00:09:40.307686 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0620 00:09:40.307732 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417}
W20260620 00:09:40.307999 15746 catalog_manager.cc:4584] TS 50bf521ab90d423eb1f0b0c9661b8161 (127.0.98.1:39261): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7381663ac53b484c9de17dee81b1bb93: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39261: connect: Connection refused (error 111)
I20260620 00:09:40.312964 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 16089
I20260620 00:09:40.317763 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 15727
I20260620 00:10:00.323269 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33689
--webserver_interface=127.0.98.62
--webserver_port=40669
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:10:00.405298 16259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:10:00.405556 16259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:10:00.405581 16259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:10:00.406705 16259 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 00:10:00.406744 16259 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 00:10:00.406757 16259 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 00:10:00.406769 16259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:10:00.406782 16259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:10:00.406795 16259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:10:00.406807 16259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:10:00.406819 16259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:10:00.408335 16259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:10:00.408528 16259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:10:00.408699 16259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:10:00.417577 16265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:00.417654 16259 server_base.cc:1034] running on GCE node
W20260620 00:10:00.417548 16268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:00.417577 16266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:00.418255 16259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:10:00.418545 16259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:10:00.419690 16259 hybrid_clock.cc:648] HybridClock initialized: now 1781914200419667 us; error 38 us; skew 500 ppm
Jun 20 00:10:00 dist-test-slave-grqt krb5kdc[15714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:10:00.422439 16259 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:10:00.422955 16259 webserver.cc:460] Webserver started at http://127.0.98.62:40669/ using document root <none> and password file <none>
I20260620 00:10:00.423197 16259 fs_manager.cc:359] Metadata directory not provided
I20260620 00:10:00.423278 16259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:10:00.424688 16259 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:10:00.425418 16275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:00.425597 16259 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:10:00.425704 16259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "136468519525499ab7998577528f32b0"
format_stamp: "Formatted at 2026-06-20 00:09:39 on dist-test-slave-grqt"
server_key: "65b8edbc516ba838cdcc3be23b7bf168"
server_key_iv: "ea93e8fb0e8c9ee5b9edf39a19898c7f"
server_key_version: "encryptionkey@0"
I20260620 00:10:00.426134 16259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:10:00.448050 16259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:10:00.450605 16259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:10:00.450784 16259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:10:00.455238 16259 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33689
I20260620 00:10:00.455293 16327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33689 every 8 connection(s)
I20260620 00:10:00.455736 16259 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:10:00.457252 16328 sys_catalog.cc:263] Verifying existing consensus state
I20260620 00:10:00.458002 16328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0: Bootstrap starting.
I20260620 00:10:00.458566 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 16259
I20260620 00:10:00.458856 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39261
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33689
--webserver_port=44353
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:10:00.467236 16328 log.cc:824] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0: Log is configured to *not* fsync() on all Append() calls
I20260620 00:10:00.469295 16328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 00:10:00.469635 16328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0: Bootstrap complete.
I20260620 00:10:00.471936 16328 raft_consensus.cc:348] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136468519525499ab7998577528f32b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33689 } }
I20260620 00:10:00.472160 16328 raft_consensus.cc:729] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136468519525499ab7998577528f32b0, State: Initialized, Role: FOLLOWER
I20260620 00:10:00.472379 16328 consensus_queue.cc:260] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136468519525499ab7998577528f32b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33689 } }
I20260620 00:10:00.472446 16328 raft_consensus.cc:388] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:10:00.472502 16328 raft_consensus.cc:482] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:10:00.472551 16328 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [term 1 FOLLOWER]: Advancing to term 2
I20260620 00:10:00.473635 16328 raft_consensus.cc:504] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136468519525499ab7998577528f32b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33689 } }
I20260620 00:10:00.473721 16328 leader_election.cc:302] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 136468519525499ab7998577528f32b0; no voters:
I20260620 00:10:00.473956 16328 leader_election.cc:288] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 00:10:00.473991 16332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 00:10:00.474169 16332 raft_consensus.cc:686] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [term 2 LEADER]: Becoming Leader. State: Replica: 136468519525499ab7998577528f32b0, State: Running, Role: LEADER
I20260620 00:10:00.474236 16328 sys_catalog.cc:564] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:10:00.474365 16332 consensus_queue.cc:237] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136468519525499ab7998577528f32b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33689 } }
I20260620 00:10:00.474664 16332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 136468519525499ab7998577528f32b0. Latest consensus state: current_term: 2 leader_uuid: "136468519525499ab7998577528f32b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136468519525499ab7998577528f32b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33689 } } }
I20260620 00:10:00.474669 16334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "136468519525499ab7998577528f32b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136468519525499ab7998577528f32b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33689 } } }
I20260620 00:10:00.474877 16334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [sys.catalog]: This master's current role is: LEADER
I20260620 00:10:00.474864 16332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [sys.catalog]: This master's current role is: LEADER
I20260620 00:10:00.475090 16341 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:10:00.475955 16341 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:10:00.476080 16341 catalog_manager.cc:1259] Loaded cluster ID: 6de8495070104f8ea90f0bd0a936c268
I20260620 00:10:00.476152 16341 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:10:00.477485 16341 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:10:00.477669 16341 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0: Loaded TSK: 0
I20260620 00:10:00.491950 16341 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0: Generated new TSK 1
I20260620 00:10:00.492087 16341 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 00:10:00.548151 16330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:10:00.548377 16330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:10:00.548403 16330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:10:00.549687 16330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:10:00.549731 16330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:10:00.549758 16330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:10:00.551306 16330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39261
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44353
--tserver_master_addrs=127.0.98.62:33689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:10:00.551568 16330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:10:00.551841 16330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:10:00.557034 16358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:00.557010 16355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:00.557296 16330 server_base.cc:1034] running on GCE node
W20260620 00:10:00.557466 16356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:00.557687 16330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:10:00.557994 16330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:10:00.559195 16330 hybrid_clock.cc:648] HybridClock initialized: now 1781914200559181 us; error 30 us; skew 500 ppm
Jun 20 00:10:00 dist-test-slave-grqt krb5kdc[15714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:10:00.562502 16330 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:10:00.563033 16330 webserver.cc:460] Webserver started at http://127.0.98.1:44353/ using document root <none> and password file <none>
I20260620 00:10:00.563274 16330 fs_manager.cc:359] Metadata directory not provided
I20260620 00:10:00.563359 16330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:10:00.564859 16330 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:10:00.565629 16365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:00.565828 16330 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:10:00.565941 16330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/wal
uuid: "50bf521ab90d423eb1f0b0c9661b8161"
format_stamp: "Formatted at 2026-06-20 00:09:39 on dist-test-slave-grqt"
server_key: "38a5349f361d02332db20f36b73990f3"
server_key_iv: "0cbc37e16844182871a9462422c0e888"
server_key_version: "encryptionkey@0"
I20260620 00:10:00.566311 16330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:10:00.583292 16330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:10:00.585826 16330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:10:00.585978 16330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:10:00.586258 16330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:10:00.586834 16372 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 00:10:00.588809 16330 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 00:10:00.588888 16330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 00:10:00.588939 16330 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 00:10:00.589748 16330 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 00:10:00.589807 16330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260620 00:10:00.589874 16372 tablet_bootstrap.cc:492] T 7381663ac53b484c9de17dee81b1bb93 P 50bf521ab90d423eb1f0b0c9661b8161: Bootstrap starting.
I20260620 00:10:00.597087 16330 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39261
I20260620 00:10:00.597121 16479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39261 every 8 connection(s)
I20260620 00:10:00.597679 16330 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:10:00.601184 16372 log.cc:824] T 7381663ac53b484c9de17dee81b1bb93 P 50bf521ab90d423eb1f0b0c9661b8161: Log is configured to *not* fsync() on all Append() calls
I20260620 00:10:00.603897 16372 tablet_bootstrap.cc:492] T 7381663ac53b484c9de17dee81b1bb93 P 50bf521ab90d423eb1f0b0c9661b8161: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 00:10:00.604218 16372 tablet_bootstrap.cc:492] T 7381663ac53b484c9de17dee81b1bb93 P 50bf521ab90d423eb1f0b0c9661b8161: Bootstrap complete.
I20260620 00:10:00.604374 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 16330
I20260620 00:10:00.604638 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46097
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33689
--webserver_port=45341
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:10:00.605507 16372 ts_tablet_manager.cc:1397] T 7381663ac53b484c9de17dee81b1bb93 P 50bf521ab90d423eb1f0b0c9661b8161: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.015s
I20260620 00:10:00.606961 16372 raft_consensus.cc:348] T 7381663ac53b484c9de17dee81b1bb93 P 50bf521ab90d423eb1f0b0c9661b8161 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49efdd3061be4caeb77db9887e9456f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40275 } } peers { permanent_uuid: "cbce6b11c84e4a0faefed80f8db30eaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46097 } } peers { permanent_uuid: "50bf521ab90d423eb1f0b0c9661b8161" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39261 } }
I20260620 00:10:00.607313 16372 raft_consensus.cc:729] T 7381663ac53b484c9de17dee81b1bb93 P 50bf521ab90d423eb1f0b0c9661b8161 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50bf521ab90d423eb1f0b0c9661b8161, State: Initialized, Role: FOLLOWER
I20260620 00:10:00.607457 16372 consensus_queue.cc:260] T 7381663ac53b484c9de17dee81b1bb93 P 50bf521ab90d423eb1f0b0c9661b8161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49efdd3061be4caeb77db9887e9456f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40275 } } peers { permanent_uuid: "cbce6b11c84e4a0faefed80f8db30eaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46097 } } peers { permanent_uuid: "50bf521ab90d423eb1f0b0c9661b8161" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39261 } }
I20260620 00:10:00.607769 16372 ts_tablet_manager.cc:1428] T 7381663ac53b484c9de17dee81b1bb93 P 50bf521ab90d423eb1f0b0c9661b8161: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
Jun 20 00:10:00 dist-test-slave-grqt krb5kdc[15714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:10:00.613433 16483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:00.598603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51089 (local address 127.0.98.62:33689)
0620 00:10:00.601772 (+ 3169us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:00.601780 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:00.601850 (+ 70us) server_negotiation.cc:407] Connection header received
0620 00:10:00.602002 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:00.602009 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:00.602070 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:00.602235 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:00.603146 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:00.604512 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:00.606142 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:00.606759 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:00.608277 (+ 1518us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:00.609521 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:00.609526 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:00.609526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:10:00.609538 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:00.611251 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:00.611797 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:00.611800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:00.611801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:00.611939 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:00.612277 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:00.612278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:00.612279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:00.612601 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:00.612685 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:00.613068 (+ 383us) server_negotiation.cc:299] Negotiation successful
0620 00:10:00.613198 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3072,"thread_start_us":92,"threads_started":1}
I20260620 00:10:00.613448 16482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:00.598480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.1:51089)
0620 00:10:00.598677 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:00.598774 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:00.599260 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:00.602313 (+ 3053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:00.602318 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:00.602666 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:10:00.602964 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:00.602974 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:00.604657 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:00.604658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:00.606012 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:00.606015 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:00.606873 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:00.606874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:00.607017 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:00.607676 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:10:00.607686 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:10:00.609359 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:10:00.611367 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:10:00.611375 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:10:00.611377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:10:00.611684 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:10:00.612039 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:10:00.612040 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:10:00.612041 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:10:00.612197 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:10:00.612715 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:10:00.612721 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:10:00.612890 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 00:10:00.613010 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 00:10:00.613199 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":94,"threads_started":1}
I20260620 00:10:00.614504 16480 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33689
I20260620 00:10:00.614630 16480 heartbeater.cc:464] Registering TS with master...
I20260620 00:10:00.614820 16480 heartbeater.cc:511] Master 127.0.98.62:33689 requested a full tablet report, sending...
I20260620 00:10:00.615525 16292 ts_manager.cc:194] Registered new tserver with Master: 50bf521ab90d423eb1f0b0c9661b8161 (127.0.98.1:39261)
I20260620 00:10:00.616534 16292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51089
I20260620 00:10:00.617491 16480 heartbeater.cc:503] Master 127.0.98.62:33689 was elected leader, sending a full tablet report...
I20260620 00:10:00.622224 16487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:00.616126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39261 (local address 127.0.0.1:38906)
0620 00:10:00.617491 (+ 1365us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:00.617531 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:00.617601 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:00.618870 (+ 1269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:00.618874 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:00.618876 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:10:00.618932 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:00.618934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:00.620252 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:00.620253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:00.621027 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:00.621029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:00.621992 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:00.621993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:00.622054 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:00.622065 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 00:10:00.622125 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 00:10:00.622164 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1332,"thread_start_us":93,"threads_started":1}
I20260620 00:10:00.622370 16488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:00.616197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38906 (local address 127.0.98.1:39261)
0620 00:10:00.618682 (+ 2485us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:00.618687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:00.618699 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:10:00.618735 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:00.618737 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:00.618752 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:00.618854 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:10:00.619012 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:00.620153 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:00.621111 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:00.621907 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:00.621984 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:00.622039 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:00.622220 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 00:10:00.622269 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2451,"thread_start_us":71,"threads_started":1}
I20260620 00:10:00.622848 16414 tablet_service.cc:1514] Processing DeleteTablet for tablet 7381663ac53b484c9de17dee81b1bb93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:09:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38906
I20260620 00:10:00.623461 16489 tablet_replica.cc:330] stopping tablet replica
I20260620 00:10:00.623574 16489 raft_consensus.cc:2229] T 7381663ac53b484c9de17dee81b1bb93 P 50bf521ab90d423eb1f0b0c9661b8161 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:10:00.623642 16489 raft_consensus.cc:2258] T 7381663ac53b484c9de17dee81b1bb93 P 50bf521ab90d423eb1f0b0c9661b8161 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:10:00.623960 16489 ts_tablet_manager.cc:1905] T 7381663ac53b484c9de17dee81b1bb93 P 50bf521ab90d423eb1f0b0c9661b8161: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:10:00.626111 16489 ts_tablet_manager.cc:1918] T 7381663ac53b484c9de17dee81b1bb93 P 50bf521ab90d423eb1f0b0c9661b8161: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:10:00.626174 16489 log.cc:1196] T 7381663ac53b484c9de17dee81b1bb93 P 50bf521ab90d423eb1f0b0c9661b8161: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/wal/wals/7381663ac53b484c9de17dee81b1bb93
I20260620 00:10:00.626421 16489 ts_tablet_manager.cc:1939] T 7381663ac53b484c9de17dee81b1bb93 P 50bf521ab90d423eb1f0b0c9661b8161: Deleting consensus metadata
I20260620 00:10:00.626914 16279 catalog_manager.cc:4841] TS 50bf521ab90d423eb1f0b0c9661b8161 (127.0.98.1:39261): tablet 7381663ac53b484c9de17dee81b1bb93 (table test-table [id=977995cb4a5e4bc1925cb56f759601fe]) successfully deleted
W20260620 00:10:00.693722 16485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:10:00.694000 16485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:10:00.694042 16485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:10:00.695261 16485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:10:00.695331 16485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:10:00.695379 16485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:10:00.696947 16485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46097
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45341
--tserver_master_addrs=127.0.98.62:33689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:10:00.697239 16485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:10:00.697507 16485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 00:10:00.703018 16485 server_base.cc:1034] running on GCE node
W20260620 00:10:00.703001 16495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:00.703004 16497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:00.702992 16494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:00.703480 16485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:10:00.703758 16485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:10:00.704952 16485 hybrid_clock.cc:648] HybridClock initialized: now 1781914200704945 us; error 36 us; skew 500 ppm
Jun 20 00:10:00 dist-test-slave-grqt krb5kdc[15714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:10:00.707669 16485 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:10:00.708086 16485 webserver.cc:460] Webserver started at http://127.0.98.2:45341/ using document root <none> and password file <none>
I20260620 00:10:00.708331 16485 fs_manager.cc:359] Metadata directory not provided
I20260620 00:10:00.708417 16485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:10:00.709867 16485 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:10:00.710484 16504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:00.710700 16485 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:10:00.710814 16485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/wal
uuid: "cbce6b11c84e4a0faefed80f8db30eaa"
format_stamp: "Formatted at 2026-06-20 00:09:39 on dist-test-slave-grqt"
server_key: "6ef693e56ec2dc7e4b3e09dc550ba60f"
server_key_iv: "28a0b898c975def66c3bc3a00bc0a42f"
server_key_version: "encryptionkey@0"
I20260620 00:10:00.711189 16485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:10:00.719846 16485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:10:00.722254 16485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:10:00.722417 16485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:10:00.722658 16485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:10:00.722972 16485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:10:00.723026 16485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:00.723096 16485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:10:00.723129 16485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:00.729279 16485 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46097
I20260620 00:10:00.729343 16617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46097 every 8 connection(s)
I20260620 00:10:00.729761 16485 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:10:00.730443 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 16485
I20260620 00:10:00.730749 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40275
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33689
--webserver_port=33931
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 00:10:00 dist-test-slave-grqt krb5kdc[15714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:10:00.740432 16483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:00.730540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42913 (local address 127.0.98.62:33689)
0620 00:10:00.730618 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:00.730620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:00.730840 (+ 220us) server_negotiation.cc:407] Connection header received
0620 00:10:00.731434 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:00.731435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:00.731451 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:00.731501 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:00.732488 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:00.733173 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:00.734439 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:00.734831 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:00.734915 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:00.737066 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:00.737068 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:00.737068 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:10:00.737078 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:00.738917 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:00.739329 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:00.739331 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:00.739331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:00.739396 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:00.739692 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:00.739693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:00.739693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:00.739860 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:00.739939 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:00.740194 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 00:10:00.740293 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 00:10:00.740454 16620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:00.730527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.2:42913)
0620 00:10:00.730705 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:00.730749 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:00.731312 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:00.731573 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:00.731577 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:00.732164 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:10:00.732373 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:00.732378 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:00.733292 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:00.733293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:00.734342 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:00.734346 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:00.734931 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:00.734932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:00.735038 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:00.735612 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:10:00.735621 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:10:00.736970 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:10:00.739014 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:10:00.739020 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:10:00.739022 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:10:00.739250 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:10:00.739490 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:10:00.739491 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:10:00.739491 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:10:00.739615 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:10:00.739932 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:10:00.739937 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:10:00.740060 (+ 123us) client_negotiation.cc:769] Sending connection context
0620 00:10:00.740147 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 00:10:00.740282 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
I20260620 00:10:00.741096 16618 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33689
I20260620 00:10:00.741226 16618 heartbeater.cc:464] Registering TS with master...
I20260620 00:10:00.741472 16618 heartbeater.cc:511] Master 127.0.98.62:33689 requested a full tablet report, sending...
I20260620 00:10:00.741888 16292 ts_manager.cc:194] Registered new tserver with Master: cbce6b11c84e4a0faefed80f8db30eaa (127.0.98.2:46097)
I20260620 00:10:00.742385 16292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42913
W20260620 00:10:00.814067 16621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:10:00.814277 16621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:10:00.814302 16621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:10:00.815532 16621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:10:00.815578 16621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:10:00.815604 16621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:10:00.817009 16621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40275
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33931
--tserver_master_addrs=127.0.98.62:33689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:10:00.817209 16621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:10:00.817466 16621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:10:00.822379 16626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:00.822479 16621 server_base.cc:1034] running on GCE node
W20260620 00:10:00.822387 16629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:00.822485 16627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:00.822832 16621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:10:00.823067 16621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:10:00.824227 16621 hybrid_clock.cc:648] HybridClock initialized: now 1781914200824230 us; error 41 us; skew 500 ppm
Jun 20 00:10:00 dist-test-slave-grqt krb5kdc[15714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:10:00.826622 16621 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:10:00.827082 16621 webserver.cc:460] Webserver started at http://127.0.98.3:33931/ using document root <none> and password file <none>
I20260620 00:10:00.827322 16621 fs_manager.cc:359] Metadata directory not provided
I20260620 00:10:00.827404 16621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:10:00.828763 16621 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:10:00.829361 16636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:00.829586 16621 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:10:00.829689 16621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/wal
uuid: "49efdd3061be4caeb77db9887e9456f6"
format_stamp: "Formatted at 2026-06-20 00:09:39 on dist-test-slave-grqt"
server_key: "589538078972f151c4c06ec4b247b957"
server_key_iv: "2fafc5451917ca96272979c9703da4f2"
server_key_version: "encryptionkey@0"
I20260620 00:10:00.830058 16621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:10:00.849141 16621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:10:00.851287 16621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:10:00.851464 16621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:10:00.851706 16621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:10:00.852030 16621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:10:00.852085 16621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:00.852120 16621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:10:00.852147 16621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:00.858126 16621 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40275
I20260620 00:10:00.858175 16749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40275 every 8 connection(s)
I20260620 00:10:00.858522 16621 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 00:10:00 dist-test-slave-grqt krb5kdc[15714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:10:00.866145 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 16621
I20260620 00:10:00.868656 16483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:00.859266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33327 (local address 127.0.98.62:33689)
0620 00:10:00.859337 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:00.859339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:00.859582 (+ 243us) server_negotiation.cc:407] Connection header received
0620 00:10:00.860070 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:00.860071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:00.860081 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:00.860130 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:00.861085 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:00.861667 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:00.862646 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:00.862845 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:00.862897 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:00.865080 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:00.865082 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:00.865082 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:10:00.865092 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:00.867030 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:00.867575 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:00.867576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:00.867576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:00.867630 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:00.867927 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:00.867928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:00.867928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:00.868052 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:00.868122 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:00.868487 (+ 365us) server_negotiation.cc:299] Negotiation successful
0620 00:10:00.868576 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 00:10:00.868943 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:00.859245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.3:33327)
0620 00:10:00.859414 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:00.859469 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:00.859975 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:00.860191 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:00.860195 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:00.860796 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:10:00.860997 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:00.861003 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:00.861748 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:00.861748 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:00.862561 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:00.862563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:00.862917 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:00.862917 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:00.863017 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:00.863586 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:10:00.863595 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:10:00.864974 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:10:00.867121 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:10:00.867129 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:10:00.867133 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:10:00.867489 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:10:00.867694 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:10:00.867694 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:10:00.867695 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:10:00.867864 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:10:00.868144 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:10:00.868150 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:10:00.868287 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 00:10:00.868459 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 00:10:00.868703 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":97,"threads_started":1}
I20260620 00:10:00.869544 16750 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33689
I20260620 00:10:00.869670 16750 heartbeater.cc:464] Registering TS with master...
I20260620 00:10:00.869851 16750 heartbeater.cc:511] Master 127.0.98.62:33689 requested a full tablet report, sending...
I20260620 00:10:00.870293 16292 ts_manager.cc:194] Registered new tserver with Master: 49efdd3061be4caeb77db9887e9456f6 (127.0.98.3:40275)
I20260620 00:10:00.870667 16292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33327
I20260620 00:10:00.873508 16754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:00.866584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52424 (local address 127.0.98.62:33689)
0620 00:10:00.866762 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:00.866764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:00.866794 (+ 30us) server_negotiation.cc:407] Connection header received
0620 00:10:00.866857 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:00.866858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:00.866870 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:00.866945 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:00.867762 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:00.868478 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:00.869577 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:00.869835 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:00.869889 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:00.871247 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:00.871248 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:00.871248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:10:00.871255 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:00.872528 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:00.872859 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:00.872860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:00.872860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:00.872907 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:00.873036 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:00.873037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:00.873037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:00.873130 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:00.873242 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:00.873308 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 00:10:00.873417 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":91,"threads_started":1}
I20260620 00:10:00.874346 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:10:00.876955 16754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:00.874739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52434 (local address 127.0.98.62:33689)
0620 00:10:00.874779 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:00.874781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:00.874886 (+ 105us) server_negotiation.cc:407] Connection header received
0620 00:10:00.874943 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:00.874944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:00.874970 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:00.875022 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:10:00.875095 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:00.875667 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:00.876368 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:00.876544 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:00.876596 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:00.876689 (+ 93us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:10:00.876777 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:10:00.876829 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:00.876844 (+ 15us) server_negotiation.cc:299] Negotiation successful
0620 00:10:00.876860 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 00:10:00.877209 16292 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:52434:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:10:00.878806 16292 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:10:00.881083 16414 tablet_service.cc:1467] Processing CreateTablet for tablet ddfb6ec02d084dbcb21f46a4c65fb5ae (DEFAULT_TABLE table=test-table [id=37493aab9e0a487fb05ffeb9a638f4b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:10:00.881307 16414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddfb6ec02d084dbcb21f46a4c65fb5ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:10:00.883028 16372 tablet_bootstrap.cc:492] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 50bf521ab90d423eb1f0b0c9661b8161: Bootstrap starting.
I20260620 00:10:00.883463 16372 tablet_bootstrap.cc:654] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 50bf521ab90d423eb1f0b0c9661b8161: Neither blocks nor log segments found. Creating new log.
I20260620 00:10:00.884199 16372 tablet_bootstrap.cc:492] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 50bf521ab90d423eb1f0b0c9661b8161: No bootstrap required, opened a new log
I20260620 00:10:00.884266 16372 ts_tablet_manager.cc:1397] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 50bf521ab90d423eb1f0b0c9661b8161: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 00:10:00.884421 16372 raft_consensus.cc:348] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 50bf521ab90d423eb1f0b0c9661b8161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49efdd3061be4caeb77db9887e9456f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40275 } } peers { permanent_uuid: "cbce6b11c84e4a0faefed80f8db30eaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46097 } } peers { permanent_uuid: "50bf521ab90d423eb1f0b0c9661b8161" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39261 } }
I20260620 00:10:00.884496 16372 raft_consensus.cc:374] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 50bf521ab90d423eb1f0b0c9661b8161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:10:00.884550 16372 raft_consensus.cc:729] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 50bf521ab90d423eb1f0b0c9661b8161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50bf521ab90d423eb1f0b0c9661b8161, State: Initialized, Role: FOLLOWER
I20260620 00:10:00.884604 16372 consensus_queue.cc:260] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 50bf521ab90d423eb1f0b0c9661b8161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49efdd3061be4caeb77db9887e9456f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40275 } } peers { permanent_uuid: "cbce6b11c84e4a0faefed80f8db30eaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46097 } } peers { permanent_uuid: "50bf521ab90d423eb1f0b0c9661b8161" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39261 } }
I20260620 00:10:00.884691 16487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:00.880656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46097 (local address 127.0.0.1:54022)
0620 00:10:00.880704 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:00.880715 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:00.880753 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:00.881521 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:00.881522 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:00.881524 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:10:00.881589 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:00.881591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:00.882458 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:00.882459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:00.883350 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:00.883353 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:00.884416 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:00.884417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:00.884480 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:00.884485 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:10:00.884554 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 00:10:00.884616 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":22}
I20260620 00:10:00.884701 16372 ts_tablet_manager.cc:1428] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 50bf521ab90d423eb1f0b0c9661b8161: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 00:10:00.884729 16758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:00.880789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54022 (local address 127.0.98.2:46097)
0620 00:10:00.881325 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:00.881329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:00.881338 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:10:00.881426 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:00.881428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:00.881445 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:00.881523 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:10:00.881659 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:00.882368 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:00.883447 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:00.884341 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:00.884415 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:00.884476 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:00.884575 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 00:10:00.884636 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":101,"threads_started":1}
I20260620 00:10:00.885454 16756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:00.880666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40275 (local address 127.0.0.1:58610)
0620 00:10:00.880849 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:00.880859 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:00.880895 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:00.881176 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:00.881177 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:00.881180 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:10:00.881225 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:00.881226 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:00.882510 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:00.882512 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:00.883547 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:00.883550 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:00.885152 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:00.885153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:00.885218 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:00.885221 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:10:00.885313 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 00:10:00.885357 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":93,"threads_started":1}
I20260620 00:10:00.885480 16757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:00.880725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58610 (local address 127.0.98.3:40275)
0620 00:10:00.881010 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:00.881014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:00.881023 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:10:00.881056 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:00.881057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:00.881069 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:00.881154 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:10:00.881326 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:00.882408 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:00.883645 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:00.884341 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:00.884415 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:00.884476 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:00.885337 (+ 861us) server_negotiation.cc:299] Negotiation successful
0620 00:10:00.885402 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":130,"threads_started":1}
I20260620 00:10:00.885437 16552 tablet_service.cc:1467] Processing CreateTablet for tablet ddfb6ec02d084dbcb21f46a4c65fb5ae (DEFAULT_TABLE table=test-table [id=37493aab9e0a487fb05ffeb9a638f4b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:10:00.885833 16552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddfb6ec02d084dbcb21f46a4c65fb5ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:10:00.886399 16684 tablet_service.cc:1467] Processing CreateTablet for tablet ddfb6ec02d084dbcb21f46a4c65fb5ae (DEFAULT_TABLE table=test-table [id=37493aab9e0a487fb05ffeb9a638f4b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:10:00.886730 16684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddfb6ec02d084dbcb21f46a4c65fb5ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:10:00.888660 16760 tablet_bootstrap.cc:492] T ddfb6ec02d084dbcb21f46a4c65fb5ae P cbce6b11c84e4a0faefed80f8db30eaa: Bootstrap starting.
I20260620 00:10:00.889413 16760 tablet_bootstrap.cc:654] T ddfb6ec02d084dbcb21f46a4c65fb5ae P cbce6b11c84e4a0faefed80f8db30eaa: Neither blocks nor log segments found. Creating new log.
I20260620 00:10:00.889873 16761 tablet_bootstrap.cc:492] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6: Bootstrap starting.
I20260620 00:10:00.890060 16760 log.cc:824] T ddfb6ec02d084dbcb21f46a4c65fb5ae P cbce6b11c84e4a0faefed80f8db30eaa: Log is configured to *not* fsync() on all Append() calls
I20260620 00:10:00.890467 16761 tablet_bootstrap.cc:654] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6: Neither blocks nor log segments found. Creating new log.
I20260620 00:10:00.890781 16760 tablet_bootstrap.cc:492] T ddfb6ec02d084dbcb21f46a4c65fb5ae P cbce6b11c84e4a0faefed80f8db30eaa: No bootstrap required, opened a new log
I20260620 00:10:00.890837 16760 ts_tablet_manager.cc:1397] T ddfb6ec02d084dbcb21f46a4c65fb5ae P cbce6b11c84e4a0faefed80f8db30eaa: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 00:10:00.891042 16761 log.cc:824] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6: Log is configured to *not* fsync() on all Append() calls
I20260620 00:10:00.891887 16761 tablet_bootstrap.cc:492] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6: No bootstrap required, opened a new log
I20260620 00:10:00.891947 16761 ts_tablet_manager.cc:1397] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:10:00.892565 16760 raft_consensus.cc:348] T ddfb6ec02d084dbcb21f46a4c65fb5ae P cbce6b11c84e4a0faefed80f8db30eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49efdd3061be4caeb77db9887e9456f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40275 } } peers { permanent_uuid: "cbce6b11c84e4a0faefed80f8db30eaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46097 } } peers { permanent_uuid: "50bf521ab90d423eb1f0b0c9661b8161" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39261 } }
I20260620 00:10:00.892750 16760 raft_consensus.cc:374] T ddfb6ec02d084dbcb21f46a4c65fb5ae P cbce6b11c84e4a0faefed80f8db30eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:10:00.892796 16760 raft_consensus.cc:729] T ddfb6ec02d084dbcb21f46a4c65fb5ae P cbce6b11c84e4a0faefed80f8db30eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbce6b11c84e4a0faefed80f8db30eaa, State: Initialized, Role: FOLLOWER
I20260620 00:10:00.892917 16760 consensus_queue.cc:260] T ddfb6ec02d084dbcb21f46a4c65fb5ae P cbce6b11c84e4a0faefed80f8db30eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49efdd3061be4caeb77db9887e9456f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40275 } } peers { permanent_uuid: "cbce6b11c84e4a0faefed80f8db30eaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46097 } } peers { permanent_uuid: "50bf521ab90d423eb1f0b0c9661b8161" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39261 } }
I20260620 00:10:00.893229 16618 heartbeater.cc:503] Master 127.0.98.62:33689 was elected leader, sending a full tablet report...
I20260620 00:10:00.893555 16760 ts_tablet_manager.cc:1428] T ddfb6ec02d084dbcb21f46a4c65fb5ae P cbce6b11c84e4a0faefed80f8db30eaa: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 00:10:00.893536 16761 raft_consensus.cc:348] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49efdd3061be4caeb77db9887e9456f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40275 } } peers { permanent_uuid: "cbce6b11c84e4a0faefed80f8db30eaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46097 } } peers { permanent_uuid: "50bf521ab90d423eb1f0b0c9661b8161" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39261 } }
I20260620 00:10:00.893733 16761 raft_consensus.cc:374] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:10:00.893783 16761 raft_consensus.cc:729] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49efdd3061be4caeb77db9887e9456f6, State: Initialized, Role: FOLLOWER
I20260620 00:10:00.893908 16761 consensus_queue.cc:260] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49efdd3061be4caeb77db9887e9456f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40275 } } peers { permanent_uuid: "cbce6b11c84e4a0faefed80f8db30eaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46097 } } peers { permanent_uuid: "50bf521ab90d423eb1f0b0c9661b8161" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39261 } }
I20260620 00:10:00.894200 16761 ts_tablet_manager.cc:1428] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 00:10:00.894268 16750 heartbeater.cc:503] Master 127.0.98.62:33689 was elected leader, sending a full tablet report...
W20260620 00:10:00.980355 16619 tablet.cc:2369] T ddfb6ec02d084dbcb21f46a4c65fb5ae P cbce6b11c84e4a0faefed80f8db30eaa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 00:10:01.102761 16481 tablet.cc:2369] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 50bf521ab90d423eb1f0b0c9661b8161: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 00:10:01.109100 16751 tablet.cc:2369] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:10:01.280575 16765 raft_consensus.cc:482] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:10:01.280702 16765 raft_consensus.cc:504] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49efdd3061be4caeb77db9887e9456f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40275 } } peers { permanent_uuid: "cbce6b11c84e4a0faefed80f8db30eaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46097 } } peers { permanent_uuid: "50bf521ab90d423eb1f0b0c9661b8161" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39261 } }
I20260620 00:10:01.281114 16765 leader_election.cc:288] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cbce6b11c84e4a0faefed80f8db30eaa (127.0.98.2:46097), 50bf521ab90d423eb1f0b0c9661b8161 (127.0.98.1:39261)
I20260620 00:10:01.284255 16758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:01.281465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57317 (local address 127.0.98.2:46097)
0620 00:10:01.281512 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:01.281514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:01.281694 (+ 180us) server_negotiation.cc:407] Connection header received
0620 00:10:01.281765 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:01.281767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:01.281785 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:01.281860 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:10:01.282043 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:01.282522 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:01.283520 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:01.283897 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:01.283984 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:01.283995 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:01.284153 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 00:10:01.284190 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 00:10:01.284264 16766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:01.281422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46097 (local address 127.0.98.3:57317)
0620 00:10:01.281603 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:01.281613 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:01.281700 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:01.281865 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:01.281866 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:01.281868 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:10:01.281968 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:01.281970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:01.282613 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:01.282615 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:01.283421 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:01.283422 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:01.283987 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:01.283987 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:01.284066 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:01.284070 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:10:01.284133 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 00:10:01.284176 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":28,"thread_start_us":109,"threads_started":1}
I20260620 00:10:01.284427 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:01.281368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39261 (local address 127.0.98.3:50431)
0620 00:10:01.281434 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:01.281447 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:01.281501 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:01.281871 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:01.281872 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:01.281874 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:10:01.281961 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:01.281963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:01.282695 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:01.282696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:01.283424 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:01.283426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:01.284132 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:01.284133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:01.284216 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:01.284219 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:10:01.284316 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 00:10:01.284356 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 00:10:01.284446 16767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:01.281488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50431 (local address 127.0.98.1:39261)
0620 00:10:01.281679 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:01.281681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:01.281691 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:10:01.281717 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:01.281718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:01.281761 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:01.281868 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:10:01.282039 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:01.282526 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:01.283523 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:01.283946 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:01.284119 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:01.284137 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:01.284329 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 00:10:01.284376 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":100,"threads_started":1}
I20260620 00:10:01.284610 16572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddfb6ec02d084dbcb21f46a4c65fb5ae" candidate_uuid: "49efdd3061be4caeb77db9887e9456f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbce6b11c84e4a0faefed80f8db30eaa" is_pre_election: true
I20260620 00:10:01.284812 16572 raft_consensus.cc:2454] T ddfb6ec02d084dbcb21f46a4c65fb5ae P cbce6b11c84e4a0faefed80f8db30eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49efdd3061be4caeb77db9887e9456f6 in term 0.
I20260620 00:10:01.284866 16434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddfb6ec02d084dbcb21f46a4c65fb5ae" candidate_uuid: "49efdd3061be4caeb77db9887e9456f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50bf521ab90d423eb1f0b0c9661b8161" is_pre_election: true
I20260620 00:10:01.285022 16434 raft_consensus.cc:2454] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 50bf521ab90d423eb1f0b0c9661b8161 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49efdd3061be4caeb77db9887e9456f6 in term 0.
I20260620 00:10:01.285053 16637 leader_election.cc:302] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49efdd3061be4caeb77db9887e9456f6, cbce6b11c84e4a0faefed80f8db30eaa; no voters:
I20260620 00:10:01.285216 16765 raft_consensus.cc:2781] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:10:01.285295 16765 raft_consensus.cc:482] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:10:01.285341 16765 raft_consensus.cc:3037] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:10:01.286089 16765 raft_consensus.cc:504] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49efdd3061be4caeb77db9887e9456f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40275 } } peers { permanent_uuid: "cbce6b11c84e4a0faefed80f8db30eaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46097 } } peers { permanent_uuid: "50bf521ab90d423eb1f0b0c9661b8161" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39261 } }
I20260620 00:10:01.286240 16765 leader_election.cc:288] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6 [CANDIDATE]: Term 1 election: Requested vote from peers cbce6b11c84e4a0faefed80f8db30eaa (127.0.98.2:46097), 50bf521ab90d423eb1f0b0c9661b8161 (127.0.98.1:39261)
I20260620 00:10:01.286419 16572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddfb6ec02d084dbcb21f46a4c65fb5ae" candidate_uuid: "49efdd3061be4caeb77db9887e9456f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbce6b11c84e4a0faefed80f8db30eaa"
I20260620 00:10:01.286403 16434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddfb6ec02d084dbcb21f46a4c65fb5ae" candidate_uuid: "49efdd3061be4caeb77db9887e9456f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50bf521ab90d423eb1f0b0c9661b8161"
I20260620 00:10:01.286500 16434 raft_consensus.cc:3037] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 50bf521ab90d423eb1f0b0c9661b8161 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:10:01.286499 16572 raft_consensus.cc:3037] T ddfb6ec02d084dbcb21f46a4c65fb5ae P cbce6b11c84e4a0faefed80f8db30eaa [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:10:01.287376 16434 raft_consensus.cc:2454] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 50bf521ab90d423eb1f0b0c9661b8161 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49efdd3061be4caeb77db9887e9456f6 in term 1.
I20260620 00:10:01.287376 16572 raft_consensus.cc:2454] T ddfb6ec02d084dbcb21f46a4c65fb5ae P cbce6b11c84e4a0faefed80f8db30eaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49efdd3061be4caeb77db9887e9456f6 in term 1.
I20260620 00:10:01.287566 16640 leader_election.cc:302] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49efdd3061be4caeb77db9887e9456f6, 50bf521ab90d423eb1f0b0c9661b8161; no voters:
I20260620 00:10:01.287675 16765 raft_consensus.cc:2781] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:10:01.287856 16765 raft_consensus.cc:686] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6 [term 1 LEADER]: Becoming Leader. State: Replica: 49efdd3061be4caeb77db9887e9456f6, State: Running, Role: LEADER
I20260620 00:10:01.287967 16765 consensus_queue.cc:237] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49efdd3061be4caeb77db9887e9456f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40275 } } peers { permanent_uuid: "cbce6b11c84e4a0faefed80f8db30eaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46097 } } peers { permanent_uuid: "50bf521ab90d423eb1f0b0c9661b8161" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39261 } }
I20260620 00:10:01.288893 16291 catalog_manager.cc:5495] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49efdd3061be4caeb77db9887e9456f6 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "49efdd3061be4caeb77db9887e9456f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49efdd3061be4caeb77db9887e9456f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40275 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbce6b11c84e4a0faefed80f8db30eaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50bf521ab90d423eb1f0b0c9661b8161" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39261 } health_report { overall_health: UNKNOWN } } }
I20260620 00:10:01.329856 16757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:01.327140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58612 (local address 127.0.98.3:40275)
0620 00:10:01.327221 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:01.327223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:01.327233 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:10:01.327285 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:01.327286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:01.327306 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:01.327374 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:10:01.327690 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:01.328167 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:01.328983 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:01.329162 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:01.329213 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:01.329458 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:10:01.329517 (+ 59us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 00:10:01.329755 (+ 238us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58612: token expired
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 00:10:01.329963 16757 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58612: token expired
I20260620 00:10:01.336822 16754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:01.330128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52446 (local address 127.0.98.62:33689)
0620 00:10:01.330174 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:01.330176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:01.330184 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:10:01.330209 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:01.330210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:01.330221 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:01.330272 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:01.330787 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:01.331247 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:01.332003 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:01.332191 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:01.332404 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:01.333868 (+ 1464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:01.333869 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:01.333869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:10:01.333876 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:01.335241 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:01.335625 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:01.335626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:01.335626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:01.335697 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:01.336115 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:01.336116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:01.336116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:01.336277 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:01.336368 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:01.336482 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 00:10:01.336555 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 00:10:01.339920 16757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:01.337813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58614 (local address 127.0.98.3:40275)
0620 00:10:01.337879 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:01.337880 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:01.337888 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:10:01.337909 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:01.337910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:01.337920 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:01.337967 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:10:01.338068 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:01.338527 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:01.339196 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:01.339386 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:01.339446 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:01.339604 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:10:01.339701 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:10:01.339742 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:01.339841 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 00:10:01.339861 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 00:10:01.341830 16572 raft_consensus.cc:1261] T ddfb6ec02d084dbcb21f46a4c65fb5ae P cbce6b11c84e4a0faefed80f8db30eaa [term 1 FOLLOWER]: Refusing update from remote peer 49efdd3061be4caeb77db9887e9456f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:10:01.341830 16434 raft_consensus.cc:1261] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 50bf521ab90d423eb1f0b0c9661b8161 [term 1 FOLLOWER]: Refusing update from remote peer 49efdd3061be4caeb77db9887e9456f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:10:01.342216 16765 consensus_queue.cc:1035] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50bf521ab90d423eb1f0b0c9661b8161" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:10:01.342389 16768 consensus_queue.cc:1035] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbce6b11c84e4a0faefed80f8db30eaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:10:01.344574 16772 mvcc.cc:205] Tried to move back new op lower bound from 7298720568692936704 to 7298720568476221440. Current Snapshot: MvccSnapshot[applied={T|T < 7298720568692936704}]
I20260620 00:10:01.344795 16771 mvcc.cc:205] Tried to move back new op lower bound from 7298720568692936704 to 7298720568476221440. Current Snapshot: MvccSnapshot[applied={T|T < 7298720568692936704 or (T in {7298720568692936704})}]
I20260620 00:10:01.345600 16773 mvcc.cc:205] Tried to move back new op lower bound from 7298720568692936704 to 7298720568476221440. Current Snapshot: MvccSnapshot[applied={T|T < 7298720568692936704 or (T in {7298720568692936704})}]
I20260620 00:10:01.346614 16292 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:10:01.346722 16292 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:10:01.347455 16292 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0: Sending DeleteTablet for 3 replicas of tablet ddfb6ec02d084dbcb21f46a4c65fb5ae
I20260620 00:10:01.347780 16552 tablet_service.cc:1514] Processing DeleteTablet for tablet ddfb6ec02d084dbcb21f46a4c65fb5ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:10:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54022
I20260620 00:10:01.347810 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 16330
I20260620 00:10:01.347807 16684 tablet_service.cc:1514] Processing DeleteTablet for tablet ddfb6ec02d084dbcb21f46a4c65fb5ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:10:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58610
I20260620 00:10:01.348604 16781 tablet_replica.cc:330] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6: stopping tablet replica
I20260620 00:10:01.348714 16781 raft_consensus.cc:2229] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:10:01.348815 16781 raft_consensus.cc:2258] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:10:01.349272 16781 ts_tablet_manager.cc:1905] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:10:01.351401 16781 ts_tablet_manager.cc:1918] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:10:01.351486 16781 log.cc:1196] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/wal/wals/ddfb6ec02d084dbcb21f46a4c65fb5ae
I20260620 00:10:01.351797 16781 ts_tablet_manager.cc:1939] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6: Deleting consensus metadata
I20260620 00:10:01.352272 16279 catalog_manager.cc:4841] TS 49efdd3061be4caeb77db9887e9456f6 (127.0.98.3:40275): tablet ddfb6ec02d084dbcb21f46a4c65fb5ae (table test-table [id=37493aab9e0a487fb05ffeb9a638f4b4]) successfully deleted
W20260620 00:10:01.354770 16279 connection.cc:535] client connection to 127.0.98.1:39261 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 00:10:01.354892 16279 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 00:10:01.354923 16782 tablet_replica.cc:330] T ddfb6ec02d084dbcb21f46a4c65fb5ae P cbce6b11c84e4a0faefed80f8db30eaa: stopping tablet replica
I20260620 00:10:01.355065 16782 raft_consensus.cc:2229] T ddfb6ec02d084dbcb21f46a4c65fb5ae P cbce6b11c84e4a0faefed80f8db30eaa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:10:01.355098 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 16485
I20260620 00:10:01.356034 16756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:01.355905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39261 (local address 127.0.0.1:38910)
0620 00:10:01.355956 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:01.355989 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 00:10:01.356164 16279 catalog_manager.cc:4584] TS 50bf521ab90d423eb1f0b0c9661b8161 (127.0.98.1:39261): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ddfb6ec02d084dbcb21f46a4c65fb5ae: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39261: connect: Connection refused (error 111)
I20260620 00:10:01.359615 16756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:01.359458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46097 (local address 127.0.0.1:54028)
0620 00:10:01.359541 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:01.359578 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 00:10:01.359938 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 16621
I20260620 00:10:01.364183 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 16259
I20260620 00:10:21.369616 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33689
--webserver_interface=127.0.98.62
--webserver_port=40669
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:10:21.452032 16784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:10:21.452304 16784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:10:21.452348 16784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:10:21.453562 16784 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 00:10:21.453629 16784 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 00:10:21.453648 16784 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 00:10:21.453666 16784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:10:21.453696 16784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:10:21.453725 16784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:10:21.453750 16784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:10:21.453779 16784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:10:21.455295 16784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:10:21.455593 16784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:10:21.455837 16784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:10:21.461221 16790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:21.461342 16784 server_base.cc:1034] running on GCE node
W20260620 00:10:21.461308 16789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:21.461267 16792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:21.461774 16784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:10:21.462090 16784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:10:21.463284 16784 hybrid_clock.cc:648] HybridClock initialized: now 1781914221463271 us; error 43 us; skew 500 ppm
Jun 20 00:10:21 dist-test-slave-grqt krb5kdc[15714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:10:21.471982 16784 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:10:21.472462 16784 webserver.cc:460] Webserver started at http://127.0.98.62:40669/ using document root <none> and password file <none>
I20260620 00:10:21.472697 16784 fs_manager.cc:359] Metadata directory not provided
I20260620 00:10:21.472769 16784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:10:21.474253 16784 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:10:21.475008 16799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:21.475173 16784 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:10:21.475278 16784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "136468519525499ab7998577528f32b0"
format_stamp: "Formatted at 2026-06-20 00:09:39 on dist-test-slave-grqt"
server_key: "65b8edbc516ba838cdcc3be23b7bf168"
server_key_iv: "ea93e8fb0e8c9ee5b9edf39a19898c7f"
server_key_version: "encryptionkey@0"
I20260620 00:10:21.475688 16784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:10:21.488404 16784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:10:21.490523 16784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:10:21.490713 16784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:10:21.495018 16784 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33689
I20260620 00:10:21.495060 16851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33689 every 8 connection(s)
I20260620 00:10:21.495442 16784 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:10:21.496884 16852 sys_catalog.cc:263] Verifying existing consensus state
I20260620 00:10:21.497658 16852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0: Bootstrap starting.
I20260620 00:10:21.505134 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 16784
I20260620 00:10:21.505512 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39261
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33689
--webserver_port=44353
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:10:21.507138 16852 log.cc:824] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0: Log is configured to *not* fsync() on all Append() calls
I20260620 00:10:21.510003 16852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 00:10:21.510362 16852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0: Bootstrap complete.
I20260620 00:10:21.512673 16852 raft_consensus.cc:348] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136468519525499ab7998577528f32b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33689 } }
I20260620 00:10:21.512907 16852 raft_consensus.cc:729] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136468519525499ab7998577528f32b0, State: Initialized, Role: FOLLOWER
I20260620 00:10:21.513083 16852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136468519525499ab7998577528f32b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33689 } }
I20260620 00:10:21.513156 16852 raft_consensus.cc:388] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:10:21.513219 16852 raft_consensus.cc:482] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:10:21.513266 16852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [term 2 FOLLOWER]: Advancing to term 3
I20260620 00:10:21.514369 16852 raft_consensus.cc:504] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136468519525499ab7998577528f32b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33689 } }
I20260620 00:10:21.514463 16852 leader_election.cc:302] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 136468519525499ab7998577528f32b0; no voters:
I20260620 00:10:21.514705 16852 leader_election.cc:288] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 00:10:21.514744 16856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 00:10:21.514904 16856 raft_consensus.cc:686] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [term 3 LEADER]: Becoming Leader. State: Replica: 136468519525499ab7998577528f32b0, State: Running, Role: LEADER
I20260620 00:10:21.514994 16852 sys_catalog.cc:564] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:10:21.515129 16856 consensus_queue.cc:237] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136468519525499ab7998577528f32b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33689 } }
I20260620 00:10:21.515587 16858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "136468519525499ab7998577528f32b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136468519525499ab7998577528f32b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33689 } } }
I20260620 00:10:21.515691 16858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [sys.catalog]: This master's current role is: LEADER
I20260620 00:10:21.515949 16857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 136468519525499ab7998577528f32b0. Latest consensus state: current_term: 3 leader_uuid: "136468519525499ab7998577528f32b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136468519525499ab7998577528f32b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33689 } } }
I20260620 00:10:21.516057 16857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0 [sys.catalog]: This master's current role is: LEADER
I20260620 00:10:21.516130 16866 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:10:21.517053 16866 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:10:21.517161 16866 catalog_manager.cc:1259] Loaded cluster ID: 6de8495070104f8ea90f0bd0a936c268
I20260620 00:10:21.517195 16866 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:10:21.518389 16866 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:10:21.518570 16866 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0: Loaded TSK: 0
I20260620 00:10:21.518618 16866 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 136468519525499ab7998577528f32b0: Loaded TSK: 1
F20260620 00:10:21.518855 16866 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd9e800b9ad google::LogMessage::Fail() at ??:0
@ 0x7fd9e800e88a google::LogMessage::SendToLog() at ??:0
@ 0x7fd9e800b4de google::LogMessage::Flush() at ??:0
@ 0x7fd9e800ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd9ea778104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fd9ea775d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fd9eff691cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fd9eff6948e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fd9eff69594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fd9eff2fc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fd9eff39ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd9e9ed90d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd9e9ece954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd9ead326db start_thread at ??:0
@ 0x7fd9e6b0961f clone at ??:0
W20260620 00:10:21.607697 16854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:10:21.607949 16854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:10:21.607992 16854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:10:21.609251 16854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:10:21.609302 16854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:10:21.609330 16854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:10:21.611049 16854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39261
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44353
--tserver_master_addrs=127.0.98.62:33689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:10:21.611317 16854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:10:21.611585 16854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:10:21.616942 16896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:21.616942 16894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:21.617185 16893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:21.617228 16854 server_base.cc:1034] running on GCE node
I20260620 00:10:21.617583 16854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:10:21.617904 16854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:10:21.619099 16854 hybrid_clock.cc:648] HybridClock initialized: now 1781914221619072 us; error 42 us; skew 500 ppm
Jun 20 00:10:21 dist-test-slave-grqt krb5kdc[15714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:10:21.622452 16854 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:10:21.623057 16854 webserver.cc:460] Webserver started at http://127.0.98.1:44353/ using document root <none> and password file <none>
I20260620 00:10:21.623322 16854 fs_manager.cc:359] Metadata directory not provided
I20260620 00:10:21.623396 16854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:10:21.624879 16854 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:10:21.625700 16903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:21.625968 16854 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:10:21.626075 16854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/wal
uuid: "50bf521ab90d423eb1f0b0c9661b8161"
format_stamp: "Formatted at 2026-06-20 00:09:39 on dist-test-slave-grqt"
server_key: "38a5349f361d02332db20f36b73990f3"
server_key_iv: "0cbc37e16844182871a9462422c0e888"
server_key_version: "encryptionkey@0"
I20260620 00:10:21.626581 16854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:10:21.645491 16854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:10:21.648291 16854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:10:21.648464 16854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:10:21.648752 16854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:10:21.649310 16910 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 00:10:21.650844 16854 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 00:10:21.650905 16854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 00:10:21.650941 16854 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 00:10:21.651805 16854 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 00:10:21.651854 16854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 00:10:21.651973 16910 tablet_bootstrap.cc:492] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 50bf521ab90d423eb1f0b0c9661b8161: Bootstrap starting.
I20260620 00:10:21.659276 16854 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39261
I20260620 00:10:21.659350 17017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39261 every 8 connection(s)
I20260620 00:10:21.659776 16854 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:10:21.661625 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 16854
I20260620 00:10:21.661924 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46097
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33689
--webserver_port=45341
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:10:21.663024 16910 log.cc:824] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 50bf521ab90d423eb1f0b0c9661b8161: Log is configured to *not* fsync() on all Append() calls
I20260620 00:10:21.665987 16910 tablet_bootstrap.cc:492] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 50bf521ab90d423eb1f0b0c9661b8161: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 00:10:21.666340 16910 tablet_bootstrap.cc:492] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 50bf521ab90d423eb1f0b0c9661b8161: Bootstrap complete.
I20260620 00:10:21.667580 16910 ts_tablet_manager.cc:1397] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 50bf521ab90d423eb1f0b0c9661b8161: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.003s
I20260620 00:10:21.669121 16910 raft_consensus.cc:348] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 50bf521ab90d423eb1f0b0c9661b8161 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49efdd3061be4caeb77db9887e9456f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40275 } } peers { permanent_uuid: "cbce6b11c84e4a0faefed80f8db30eaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46097 } } peers { permanent_uuid: "50bf521ab90d423eb1f0b0c9661b8161" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39261 } }
I20260620 00:10:21.669540 16910 raft_consensus.cc:729] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 50bf521ab90d423eb1f0b0c9661b8161 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50bf521ab90d423eb1f0b0c9661b8161, State: Initialized, Role: FOLLOWER
I20260620 00:10:21.669727 16910 consensus_queue.cc:260] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 50bf521ab90d423eb1f0b0c9661b8161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49efdd3061be4caeb77db9887e9456f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40275 } } peers { permanent_uuid: "cbce6b11c84e4a0faefed80f8db30eaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46097 } } peers { permanent_uuid: "50bf521ab90d423eb1f0b0c9661b8161" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39261 } }
I20260620 00:10:21.670034 16910 ts_tablet_manager.cc:1428] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 50bf521ab90d423eb1f0b0c9661b8161: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
W20260620 00:10:21.729043 17020 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:10:21.660548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.1:36463)
0620 00:10:21.660785 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:21.660851 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:21.661761 (+ 910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:21.728788 (+ 67027us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":103,"threads_started":1}
W20260620 00:10:21.729429 17018 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33689 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33689: Client connection negotiation failed: client connection to 127.0.98.62:33689: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 00:10:21.729872 17020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:21.729701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.1:35629)
0620 00:10:21.729797 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:21.729847 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 00:10:21.730259 17020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:21.730152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.1:39761)
0620 00:10:21.730207 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:21.730233 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 00:10:21.730433 17018 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 00:10:21.750379 17021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:10:21.750617 17021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:10:21.750641 17021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:10:21.751889 17021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:10:21.751945 17021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:10:21.751981 17021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:10:21.753602 17021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46097
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45341
--tserver_master_addrs=127.0.98.62:33689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:10:21.753854 17021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:10:21.754130 17021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:10:21.759404 17031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:21.759421 17028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:21.759438 17029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:21.759824 17021 server_base.cc:1034] running on GCE node
I20260620 00:10:21.760053 17021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:10:21.760309 17021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:10:21.761433 17021 hybrid_clock.cc:648] HybridClock initialized: now 1781914221761423 us; error 40 us; skew 500 ppm
Jun 20 00:10:21 dist-test-slave-grqt krb5kdc[15714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:10:21.764007 17021 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:10:21.764443 17021 webserver.cc:460] Webserver started at http://127.0.98.2:45341/ using document root <none> and password file <none>
I20260620 00:10:21.764693 17021 fs_manager.cc:359] Metadata directory not provided
I20260620 00:10:21.764772 17021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:10:21.766203 17021 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:10:21.766939 17038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:21.767174 17021 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:10:21.767258 17021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/wal
uuid: "cbce6b11c84e4a0faefed80f8db30eaa"
format_stamp: "Formatted at 2026-06-20 00:09:39 on dist-test-slave-grqt"
server_key: "6ef693e56ec2dc7e4b3e09dc550ba60f"
server_key_iv: "28a0b898c975def66c3bc3a00bc0a42f"
server_key_version: "encryptionkey@0"
I20260620 00:10:21.767594 17021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:10:21.799783 17021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:10:21.802320 17021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:10:21.802439 17021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:10:21.802639 17021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:10:21.803169 17045 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 00:10:21.804634 17021 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 00:10:21.804688 17021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 00:10:21.804714 17021 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 00:10:21.805418 17021 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 00:10:21.805452 17021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.001s
I20260620 00:10:21.805558 17045 tablet_bootstrap.cc:492] T ddfb6ec02d084dbcb21f46a4c65fb5ae P cbce6b11c84e4a0faefed80f8db30eaa: Bootstrap starting.
I20260620 00:10:21.812119 17021 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46097
I20260620 00:10:21.812220 17152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46097 every 8 connection(s)
I20260620 00:10:21.812554 17021 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:10:21.813740 17155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:21.813246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.2:46511)
0620 00:10:21.813464 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:21.813561 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":79,"threads_started":1}
W20260620 00:10:21.813961 17153 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33689 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33689: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
I20260620 00:10:21.814271 17155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:21.814173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.2:59391)
0620 00:10:21.814224 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:21.814251 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 00:10:21.814589 17155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:21.814530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.2:39427)
0620 00:10:21.814555 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:21.814575 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260620 00:10:21.814711 17153 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 00:10:21.816437 17045 log.cc:824] T ddfb6ec02d084dbcb21f46a4c65fb5ae P cbce6b11c84e4a0faefed80f8db30eaa: Log is configured to *not* fsync() on all Append() calls
I20260620 00:10:21.817875 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 17021
I20260620 00:10:21.818135 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40275
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33689
--webserver_port=33931
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:10:21.818787 17045 tablet_bootstrap.cc:492] T ddfb6ec02d084dbcb21f46a4c65fb5ae P cbce6b11c84e4a0faefed80f8db30eaa: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 00:10:21.819051 17045 tablet_bootstrap.cc:492] T ddfb6ec02d084dbcb21f46a4c65fb5ae P cbce6b11c84e4a0faefed80f8db30eaa: Bootstrap complete.
I20260620 00:10:21.819985 17045 ts_tablet_manager.cc:1397] T ddfb6ec02d084dbcb21f46a4c65fb5ae P cbce6b11c84e4a0faefed80f8db30eaa: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.012s
I20260620 00:10:21.822283 17045 raft_consensus.cc:348] T ddfb6ec02d084dbcb21f46a4c65fb5ae P cbce6b11c84e4a0faefed80f8db30eaa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49efdd3061be4caeb77db9887e9456f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40275 } } peers { permanent_uuid: "cbce6b11c84e4a0faefed80f8db30eaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46097 } } peers { permanent_uuid: "50bf521ab90d423eb1f0b0c9661b8161" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39261 } }
I20260620 00:10:21.822762 17045 raft_consensus.cc:729] T ddfb6ec02d084dbcb21f46a4c65fb5ae P cbce6b11c84e4a0faefed80f8db30eaa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbce6b11c84e4a0faefed80f8db30eaa, State: Initialized, Role: FOLLOWER
I20260620 00:10:21.822923 17045 consensus_queue.cc:260] T ddfb6ec02d084dbcb21f46a4c65fb5ae P cbce6b11c84e4a0faefed80f8db30eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49efdd3061be4caeb77db9887e9456f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40275 } } peers { permanent_uuid: "cbce6b11c84e4a0faefed80f8db30eaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46097 } } peers { permanent_uuid: "50bf521ab90d423eb1f0b0c9661b8161" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39261 } }
I20260620 00:10:21.823226 17045 ts_tablet_manager.cc:1428] T ddfb6ec02d084dbcb21f46a4c65fb5ae P cbce6b11c84e4a0faefed80f8db30eaa: Time spent starting tablet: real 0.003s user 0.003s sys 0.003s
I20260620 00:10:21.823529 17155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:21.823413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.2:42499)
0620 00:10:21.823455 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:21.823494 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260620 00:10:21.823895 17155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:21.823780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.2:50815)
0620 00:10:21.823837 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:21.823865 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 00:10:21.900761 17157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:10:21.900966 17157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:10:21.900990 17157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:10:21.902206 17157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:10:21.902261 17157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:10:21.902288 17157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:10:21.903822 17157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40275
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33931
--tserver_master_addrs=127.0.98.62:33689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:10:21.904067 17157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:10:21.904313 17157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:10:21.909255 17163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:21.909250 17166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:21.909250 17164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:21.909359 17157 server_base.cc:1034] running on GCE node
I20260620 00:10:21.909688 17157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:10:21.909904 17157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:10:21.911069 17157 hybrid_clock.cc:648] HybridClock initialized: now 1781914221911056 us; error 30 us; skew 500 ppm
Jun 20 00:10:21 dist-test-slave-grqt krb5kdc[15714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:10:21.913800 17157 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:10:21.914333 17157 webserver.cc:460] Webserver started at http://127.0.98.3:33931/ using document root <none> and password file <none>
I20260620 00:10:21.914580 17157 fs_manager.cc:359] Metadata directory not provided
I20260620 00:10:21.914628 17157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:10:21.916152 17157 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:10:21.916760 17173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:21.916988 17157 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:10:21.917083 17157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/wal
uuid: "49efdd3061be4caeb77db9887e9456f6"
format_stamp: "Formatted at 2026-06-20 00:09:39 on dist-test-slave-grqt"
server_key: "589538078972f151c4c06ec4b247b957"
server_key_iv: "2fafc5451917ca96272979c9703da4f2"
server_key_version: "encryptionkey@0"
I20260620 00:10:21.917512 17157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:10:21.936908 17157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:10:21.938896 17157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:10:21.939066 17157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:10:21.939309 17157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:10:21.939618 17157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:10:21.939672 17157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:21.939713 17157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:10:21.939741 17157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:21.945595 17157 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40275
I20260620 00:10:21.945649 17286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40275 every 8 connection(s)
I20260620 00:10:21.945968 17157 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:10:21.947003 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:21.946628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.3:48137)
0620 00:10:21.946779 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:21.946849 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":69,"threads_started":1}
W20260620 00:10:21.947418 17287 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33689 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33689: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
I20260620 00:10:21.947674 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:21.947587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.3:39921)
0620 00:10:21.947630 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:21.947657 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 00:10:21.947939 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:21.947849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.3:40653)
0620 00:10:21.947888 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:21.947926 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 00:10:21.948047 17287 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 00:10:21.953649 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 17157
I20260620 00:10:22.731110 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:22.730857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.1:37749)
0620 00:10:22.731011 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:22.731049 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":86,"threads_started":1}
I20260620 00:10:22.824757 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:22.824421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.2:46673)
0620 00:10:22.824638 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:22.824686 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":70,"threads_started":1}
I20260620 00:10:22.909300 17293 raft_consensus.cc:482] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 50bf521ab90d423eb1f0b0c9661b8161 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:10:22.909497 17293 raft_consensus.cc:504] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 50bf521ab90d423eb1f0b0c9661b8161 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49efdd3061be4caeb77db9887e9456f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40275 } } peers { permanent_uuid: "cbce6b11c84e4a0faefed80f8db30eaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46097 } } peers { permanent_uuid: "50bf521ab90d423eb1f0b0c9661b8161" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39261 } }
I20260620 00:10:22.909888 17293 leader_election.cc:288] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 50bf521ab90d423eb1f0b0c9661b8161 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49efdd3061be4caeb77db9887e9456f6 (127.0.98.3:40275), cbce6b11c84e4a0faefed80f8db30eaa (127.0.98.2:46097)
Jun 20 00:10:22 dist-test-slave-grqt krb5kdc[15714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Jun 20 00:10:22 dist-test-slave-grqt krb5kdc[15714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260620 00:10:22.920383 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:22.910302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47057 (local address 127.0.98.3:40275)
0620 00:10:22.910506 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:22.910517 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:22.910542 (+ 25us) server_negotiation.cc:407] Connection header received
0620 00:10:22.911138 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:22.911141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:22.911218 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:22.911391 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:22.911736 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:22.912719 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:22.913546 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:22.913998 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:22.914125 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:22.916614 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:22.916618 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:22.916618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:10:22.916631 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:22.918608 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:22.918984 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:22.918987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:22.918988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:22.919090 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:22.919361 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:22.919362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:22.919362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:22.919664 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:22.919752 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:22.920114 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 00:10:22.920257 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":89,"threads_started":1}
I20260620 00:10:22.920418 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:22.910223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55335 (local address 127.0.98.2:46097)
0620 00:10:22.910393 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:22.910402 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:22.910487 (+ 85us) server_negotiation.cc:407] Connection header received
0620 00:10:22.910613 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:22.910615 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:22.910679 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:22.910819 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:22.911507 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:22.912393 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:22.913414 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:22.913822 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:22.913944 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:22.916317 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:22.916321 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:22.916322 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:10:22.916335 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:22.918428 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:22.918871 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:22.918874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:22.918875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:22.918992 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:22.919301 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:22.919302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:22.919302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:22.919665 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:22.919752 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:22.920114 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 00:10:22.920257 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
I20260620 00:10:22.920409 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:22.910102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46097 (local address 127.0.98.1:55335)
0620 00:10:22.910378 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:22.910388 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:22.910429 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:22.910884 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:22.910887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:22.911208 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:10:22.911391 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:22.911396 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:22.912504 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:22.912505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:22.913277 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:22.913279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:22.913914 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:22.913916 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:22.914027 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:22.914571 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:10:22.914580 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:10:22.916170 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:10:22.918524 (+ 2354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:10:22.918532 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:10:22.918533 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:10:22.918758 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:10:22.919068 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:10:22.919069 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:10:22.919069 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:10:22.919216 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:10:22.919741 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:10:22.919750 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:10:22.919940 (+ 190us) client_negotiation.cc:769] Sending connection context
0620 00:10:22.920041 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 00:10:22.920258 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":59,"thread_start_us":60,"threads_started":1}
I20260620 00:10:22.920485 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:22.910102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40275 (local address 127.0.98.1:47057)
0620 00:10:22.910176 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:22.910188 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:22.910262 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:22.911368 (+ 1106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:22.911369 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:22.911567 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:10:22.911617 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:22.911619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:22.912825 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:22.912826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:22.913435 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:22.913437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:22.914102 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:22.914103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:22.914164 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:22.914746 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:10:22.914752 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:10:22.916510 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:10:22.918692 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:10:22.918693 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:10:22.918694 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:10:22.918893 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:10:22.919216 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:10:22.919217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:10:22.919217 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:10:22.919309 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:10:22.919741 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:10:22.919750 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:10:22.919918 (+ 168us) client_negotiation.cc:769] Sending connection context
0620 00:10:22.920041 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 00:10:22.920350 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":141}
I20260620 00:10:22.921658 17107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddfb6ec02d084dbcb21f46a4c65fb5ae" candidate_uuid: "50bf521ab90d423eb1f0b0c9661b8161" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "cbce6b11c84e4a0faefed80f8db30eaa" is_pre_election: true
I20260620 00:10:22.921824 17107 raft_consensus.cc:2454] T ddfb6ec02d084dbcb21f46a4c65fb5ae P cbce6b11c84e4a0faefed80f8db30eaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50bf521ab90d423eb1f0b0c9661b8161 in term 1.
I20260620 00:10:22.922179 16904 leader_election.cc:302] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 50bf521ab90d423eb1f0b0c9661b8161 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50bf521ab90d423eb1f0b0c9661b8161, cbce6b11c84e4a0faefed80f8db30eaa; no voters:
I20260620 00:10:22.922312 17293 raft_consensus.cc:2781] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 50bf521ab90d423eb1f0b0c9661b8161 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 00:10:22.922377 17293 raft_consensus.cc:482] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 50bf521ab90d423eb1f0b0c9661b8161 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:10:22.922400 17293 raft_consensus.cc:3037] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 50bf521ab90d423eb1f0b0c9661b8161 [term 1 FOLLOWER]: Advancing to term 2
W20260620 00:10:22.922415 16907 leader_election.cc:341] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 50bf521ab90d423eb1f0b0c9661b8161 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 49efdd3061be4caeb77db9887e9456f6 (127.0.98.3:40275): Not found: Tablet not found: ddfb6ec02d084dbcb21f46a4c65fb5ae
I20260620 00:10:22.923590 17293 raft_consensus.cc:504] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 50bf521ab90d423eb1f0b0c9661b8161 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49efdd3061be4caeb77db9887e9456f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40275 } } peers { permanent_uuid: "cbce6b11c84e4a0faefed80f8db30eaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46097 } } peers { permanent_uuid: "50bf521ab90d423eb1f0b0c9661b8161" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39261 } }
I20260620 00:10:22.923751 17293 leader_election.cc:288] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 50bf521ab90d423eb1f0b0c9661b8161 [CANDIDATE]: Term 2 election: Requested vote from peers 49efdd3061be4caeb77db9887e9456f6 (127.0.98.3:40275), cbce6b11c84e4a0faefed80f8db30eaa (127.0.98.2:46097)
I20260620 00:10:22.923956 17107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddfb6ec02d084dbcb21f46a4c65fb5ae" candidate_uuid: "50bf521ab90d423eb1f0b0c9661b8161" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "cbce6b11c84e4a0faefed80f8db30eaa"
W20260620 00:10:22.924065 16907 leader_election.cc:341] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 50bf521ab90d423eb1f0b0c9661b8161 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 49efdd3061be4caeb77db9887e9456f6 (127.0.98.3:40275): Not found: Tablet not found: ddfb6ec02d084dbcb21f46a4c65fb5ae
I20260620 00:10:22.924054 17107 raft_consensus.cc:3037] T ddfb6ec02d084dbcb21f46a4c65fb5ae P cbce6b11c84e4a0faefed80f8db30eaa [term 1 FOLLOWER]: Advancing to term 2
I20260620 00:10:22.925165 17107 raft_consensus.cc:2454] T ddfb6ec02d084dbcb21f46a4c65fb5ae P cbce6b11c84e4a0faefed80f8db30eaa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50bf521ab90d423eb1f0b0c9661b8161 in term 2.
I20260620 00:10:22.925343 16904 leader_election.cc:302] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 50bf521ab90d423eb1f0b0c9661b8161 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 50bf521ab90d423eb1f0b0c9661b8161, cbce6b11c84e4a0faefed80f8db30eaa; no voters: 49efdd3061be4caeb77db9887e9456f6
I20260620 00:10:22.925474 17293 raft_consensus.cc:2781] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 50bf521ab90d423eb1f0b0c9661b8161 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 00:10:22.925592 17293 raft_consensus.cc:686] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 50bf521ab90d423eb1f0b0c9661b8161 [term 2 LEADER]: Becoming Leader. State: Replica: 50bf521ab90d423eb1f0b0c9661b8161, State: Running, Role: LEADER
I20260620 00:10:22.925707 17293 consensus_queue.cc:237] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 50bf521ab90d423eb1f0b0c9661b8161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49efdd3061be4caeb77db9887e9456f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40275 } } peers { permanent_uuid: "cbce6b11c84e4a0faefed80f8db30eaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46097 } } peers { permanent_uuid: "50bf521ab90d423eb1f0b0c9661b8161" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39261 } }
I20260620 00:10:22.925943 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:22.925829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.1:58885)
0620 00:10:22.925873 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:22.925917 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260620 00:10:22.948608 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:22.948371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.3:43737)
0620 00:10:22.948484 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:22.948558 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":58,"threads_started":1}
W20260620 00:10:23.368477 16907 consensus_peers.cc:487] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 50bf521ab90d423eb1f0b0c9661b8161 -> Peer 49efdd3061be4caeb77db9887e9456f6 (127.0.98.3:40275): Couldn't send request to peer 49efdd3061be4caeb77db9887e9456f6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ddfb6ec02d084dbcb21f46a4c65fb5ae. This is attempt 1: this message will repeat every 5th retry.
I20260620 00:10:23.470502 17107 raft_consensus.cc:1261] T ddfb6ec02d084dbcb21f46a4c65fb5ae P cbce6b11c84e4a0faefed80f8db30eaa [term 2 FOLLOWER]: Refusing update from remote peer 50bf521ab90d423eb1f0b0c9661b8161: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260620 00:10:23.470984 17293 consensus_queue.cc:1035] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 50bf521ab90d423eb1f0b0c9661b8161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbce6b11c84e4a0faefed80f8db30eaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 00:10:23.472002 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:23.471773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.2:41251)
0620 00:10:23.471857 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:23.471950 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71,"thread_start_us":43,"threads_started":1}
I20260620 00:10:23.472954 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:23.472763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.1:49519)
0620 00:10:23.472874 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:23.472906 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":63,"threads_started":1}
I20260620 00:10:23.473114 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:23.472994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.2:35607)
0620 00:10:23.473063 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:23.473088 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 00:10:23.900051 17307 ts_tablet_manager.cc:927] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6: Initiating tablet copy from peer 50bf521ab90d423eb1f0b0c9661b8161 (127.0.98.1:39261)
I20260620 00:10:23.900301 17307 tablet_copy_client.cc:316] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.1:39261
Jun 20 00:10:23 dist-test-slave-grqt krb5kdc[15714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 00:10:23.908195 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:23.900674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39261 (local address 127.0.98.3:53377)
0620 00:10:23.900881 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:23.900895 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:23.900988 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:23.901135 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:23.901139 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:23.901478 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:10:23.901536 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:23.901538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:23.902361 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:23.902362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:23.902965 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:23.902967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:23.903454 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:23.903455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:23.903504 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:23.903986 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:10:23.903990 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:10:23.905304 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:10:23.907152 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:10:23.907158 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:10:23.907159 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:10:23.907346 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:10:23.907530 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:10:23.907531 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:10:23.907531 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:10:23.907583 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:10:23.907890 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:10:23.907896 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:10:23.907964 (+ 68us) client_negotiation.cc:769] Sending connection context
0620 00:10:23.908033 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 00:10:23.908118 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":101,"threads_started":1}
I20260620 00:10:23.908200 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:23.900758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53377 (local address 127.0.98.1:39261)
0620 00:10:23.900937 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:23.900941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:23.900987 (+ 46us) server_negotiation.cc:407] Connection header received
0620 00:10:23.901032 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:23.901034 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:23.901047 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:23.901114 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:23.901608 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:23.902284 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:23.903067 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:23.903370 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:23.903442 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:23.905420 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:23.905427 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:23.905428 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:10:23.905434 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:23.907078 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:23.907418 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:23.907421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:23.907421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:23.907473 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:23.907657 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:23.907658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:23.907659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:23.907828 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:23.907893 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:23.908071 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 00:10:23.908134 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":79,"threads_started":1}
I20260620 00:10:23.908645 16992 tablet_copy_service.cc:140] P 50bf521ab90d423eb1f0b0c9661b8161: Received BeginTabletCopySession request for tablet ddfb6ec02d084dbcb21f46a4c65fb5ae from peer 49efdd3061be4caeb77db9887e9456f6 ({username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53377)
I20260620 00:10:23.908728 16992 tablet_copy_service.cc:161] P 50bf521ab90d423eb1f0b0c9661b8161: Beginning new tablet copy session on tablet ddfb6ec02d084dbcb21f46a4c65fb5ae from peer 49efdd3061be4caeb77db9887e9456f6 at {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53377: session id = 49efdd3061be4caeb77db9887e9456f6-ddfb6ec02d084dbcb21f46a4c65fb5ae
I20260620 00:10:23.909224 16992 tablet_copy_source_session.cc:205] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 50bf521ab90d423eb1f0b0c9661b8161: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 00:10:23.910171 17307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddfb6ec02d084dbcb21f46a4c65fb5ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:10:23.912691 17307 tablet_copy_client.cc:791] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6: tablet copy: Starting download of 0 data blocks...
I20260620 00:10:23.912844 17307 tablet_copy_client.cc:655] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6: tablet copy: Starting download of 1 WAL segments...
I20260620 00:10:23.913695 17307 tablet_copy_client.cc:531] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 00:10:23.914423 17307 tablet_bootstrap.cc:492] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6: Bootstrap starting.
I20260620 00:10:23.915823 17307 log.cc:824] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6: Log is configured to *not* fsync() on all Append() calls
I20260620 00:10:23.918277 17307 tablet_bootstrap.cc:492] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 00:10:23.918511 17307 tablet_bootstrap.cc:492] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6: Bootstrap complete.
I20260620 00:10:23.918658 17307 ts_tablet_manager.cc:1397] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260620 00:10:23.919624 17307 raft_consensus.cc:348] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49efdd3061be4caeb77db9887e9456f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40275 } } peers { permanent_uuid: "cbce6b11c84e4a0faefed80f8db30eaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46097 } } peers { permanent_uuid: "50bf521ab90d423eb1f0b0c9661b8161" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39261 } }
I20260620 00:10:23.919838 17307 raft_consensus.cc:729] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49efdd3061be4caeb77db9887e9456f6, State: Initialized, Role: FOLLOWER
I20260620 00:10:23.920063 17307 consensus_queue.cc:260] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49efdd3061be4caeb77db9887e9456f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40275 } } peers { permanent_uuid: "cbce6b11c84e4a0faefed80f8db30eaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46097 } } peers { permanent_uuid: "50bf521ab90d423eb1f0b0c9661b8161" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39261 } }
I20260620 00:10:23.920303 17307 ts_tablet_manager.cc:1428] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 00:10:23.920624 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:23.920520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.3:55323)
0620 00:10:23.920567 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:23.920597 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 00:10:23.920725 16992 tablet_copy_service.cc:342] P 50bf521ab90d423eb1f0b0c9661b8161: Request end of tablet copy session 49efdd3061be4caeb77db9887e9456f6-ddfb6ec02d084dbcb21f46a4c65fb5ae received from {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53377
I20260620 00:10:23.920787 16992 tablet_copy_service.cc:433] P 50bf521ab90d423eb1f0b0c9661b8161: ending tablet copy session 49efdd3061be4caeb77db9887e9456f6-ddfb6ec02d084dbcb21f46a4c65fb5ae on tablet ddfb6ec02d084dbcb21f46a4c65fb5ae with peer 49efdd3061be4caeb77db9887e9456f6
I20260620 00:10:23.921038 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:23.920921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.3:48977)
0620 00:10:23.920967 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:23.921012 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 00:10:24.473625 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:24.473421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.1:49571)
0620 00:10:24.473542 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:24.473580 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
I20260620 00:10:24.473696 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:24.473499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.2:45889)
0620 00:10:24.473609 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:24.473639 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":49,"threads_started":1}
I20260620 00:10:24.521646 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:24.521506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.1:43543)
0620 00:10:24.521563 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:24.521601 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 00:10:24.521677 17241 raft_consensus.cc:1203] T ddfb6ec02d084dbcb21f46a4c65fb5ae P 49efdd3061be4caeb77db9887e9456f6 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260620 00:10:24.522348 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:24.522158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.3:43243)
0620 00:10:24.522262 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:24.522295 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":70,"threads_started":1}
I20260620 00:10:25.474361 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:25.474141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.2:39803)
0620 00:10:25.474267 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:25.474311 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
I20260620 00:10:25.522292 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:25.522080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.1:59997)
0620 00:10:25.522206 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:25.522245 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":68,"threads_started":1}
I20260620 00:10:25.522861 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:25.522686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.3:52877)
0620 00:10:25.522793 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:25.522826 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":50,"threads_started":1}
I20260620 00:10:26.475183 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:26.474945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.2:47677)
0620 00:10:26.475093 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:26.475133 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":66,"threads_started":1}
I20260620 00:10:26.522954 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:26.522761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.1:60053)
0620 00:10:26.522879 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:26.522908 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":67,"threads_started":1}
I20260620 00:10:26.523425 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:26.523242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.3:40799)
0620 00:10:26.523349 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:26.523385 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":65,"threads_started":1}
I20260620 00:10:27.475881 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:27.475650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.2:40123)
0620 00:10:27.475795 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:27.475835 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":60,"threads_started":1}
I20260620 00:10:27.523733 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:27.523499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.1:53699)
0620 00:10:27.523643 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:27.523686 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":76,"threads_started":1}
I20260620 00:10:27.524084 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:27.523860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.3:59451)
0620 00:10:27.524002 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:27.524035 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":79,"threads_started":1}
I20260620 00:10:27.524142 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:27.524046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.1:33823)
0620 00:10:27.524095 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:27.524121 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 00:10:28.476545 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:28.476337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.2:39851)
0620 00:10:28.476453 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:28.476489 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":58,"threads_started":1}
I20260620 00:10:28.524837 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:28.524594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.1:35239)
0620 00:10:28.524758 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:28.524791 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":80,"threads_started":1}
I20260620 00:10:28.524869 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:28.524594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.3:48301)
0620 00:10:28.524750 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:28.524791 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":78,"threads_started":1}
I20260620 00:10:29.477165 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:29.476962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.2:57311)
0620 00:10:29.477085 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:29.477118 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
I20260620 00:10:29.525632 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:29.525351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.1:50515)
0620 00:10:29.525537 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:29.525586 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":110,"threads_started":1}
I20260620 00:10:29.525666 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:29.525415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.3:35413)
0620 00:10:29.525585 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:29.525618 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":117,"threads_started":1}
I20260620 00:10:30.477831 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:30.477621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.2:55461)
0620 00:10:30.477748 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:30.477784 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
I20260620 00:10:30.526332 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:30.526118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.3:34073)
0620 00:10:30.526253 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:30.526286 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
I20260620 00:10:30.526340 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:30.526105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.1:32843)
0620 00:10:30.526260 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:30.526297 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":67,"threads_started":1}
I20260620 00:10:31.478477 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:31.478263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.2:33377)
0620 00:10:31.478394 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:31.478430 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
I20260620 00:10:31.527073 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:31.526831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.3:35255)
0620 00:10:31.526989 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:31.527027 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":76,"threads_started":1}
I20260620 00:10:31.527073 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:31.526831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.1:57889)
0620 00:10:31.526983 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:31.527027 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":74,"threads_started":1}
I20260620 00:10:32.479270 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:32.479035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.2:56951)
0620 00:10:32.479178 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:32.479223 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":66,"threads_started":1}
I20260620 00:10:32.527832 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:32.527597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.1:42877)
0620 00:10:32.527756 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:32.527786 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":77,"threads_started":1}
I20260620 00:10:32.527843 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:32.527597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.3:58839)
0620 00:10:32.527761 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:32.527802 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":75,"threads_started":1}
I20260620 00:10:33.480105 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:33.479858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.2:56117)
0620 00:10:33.480011 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:33.480058 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":64,"threads_started":1}
I20260620 00:10:33.528637 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:33.528425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.1:47329)
0620 00:10:33.528566 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:33.528597 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
I20260620 00:10:33.528640 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:33.528386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.3:42927)
0620 00:10:33.528555 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:33.528591 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":89,"threads_started":1}
I20260620 00:10:34.481004 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:34.480738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.2:46287)
0620 00:10:34.480887 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:34.480919 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":94,"threads_started":1}
I20260620 00:10:34.529467 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:34.529168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.1:36835)
0620 00:10:34.529327 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:34.529407 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":104,"threads_started":1}
I20260620 00:10:34.529484 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:34.529171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.3:36667)
0620 00:10:34.529323 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:34.529428 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":101,"threads_started":1}
I20260620 00:10:35.481858 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:35.481624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.2:44953)
0620 00:10:35.481755 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:35.481798 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":67,"threads_started":1}
I20260620 00:10:35.530257 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:35.529994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.3:47699)
0620 00:10:35.530169 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:35.530202 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":89,"threads_started":1}
I20260620 00:10:35.530232 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:35.529994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.1:57099)
0620 00:10:35.530145 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:35.530187 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":86,"threads_started":1}
I20260620 00:10:36.482544 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:36.482337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.2:56743)
0620 00:10:36.482458 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:36.482496 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":63,"threads_started":1}
I20260620 00:10:36.531071 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:36.530822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.1:50285)
0620 00:10:36.530981 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:36.531025 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":62,"threads_started":1}
I20260620 00:10:36.531108 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:36.530867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33689 (local address 127.0.98.3:45277)
0620 00:10:36.531014 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:36.531047 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
I20260620 00:10:36.962860 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 16854
I20260620 00:10:36.967871 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 17021
I20260620 00:10:36.972540 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 17157
2026-06-20T00:10:36Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58763 ms)
[----------] 30 tests from SecurityITest (235264 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:10:37 dist-test-slave-grqt krb5kdc[17360](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:10:37 dist-test-slave-grqt krb5kdc[17360](info): set up 2 sockets
Jun 20 00:10:37 dist-test-slave-grqt krb5kdc[17360](info): commencing operation
krb5kdc: starting...
W20260620 00:10:38.017133 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:10:38 dist-test-slave-grqt krb5kdc[17360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:10:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:10:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:10:38.140185 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45859
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45859
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:10:38.243116 17373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:10:38.243371 17373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:10:38.243412 17373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:10:38.244596 17373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:10:38.244666 17373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:10:38.244694 17373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:10:38.244726 17373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:10:38.244750 17373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:10:38.246304 17373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:10:38.246590 17373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:10:38.246819 17373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:10:38.251821 17379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:38.251848 17378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:38.251835 17381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:38.251870 17373 server_base.cc:1034] running on GCE node
I20260620 00:10:38.252302 17373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:10:38.252548 17373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:10:38.253681 17373 hybrid_clock.cc:648] HybridClock initialized: now 1781914238253670 us; error 31 us; skew 500 ppm
Jun 20 00:10:38 dist-test-slave-grqt krb5kdc[17360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:10:38.256249 17373 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:10:38.256651 17373 webserver.cc:460] Webserver started at http://127.0.98.62:46469/ using document root <none> and password file <none>
I20260620 00:10:38.256842 17373 fs_manager.cc:359] Metadata directory not provided
I20260620 00:10:38.256884 17373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:10:38.256989 17373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:10:38.258100 17373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "ecd964f334a746e2b2428661db02baa8"
format_stamp: "Formatted at 2026-06-20 00:10:38 on dist-test-slave-grqt"
server_key: "2ec40870b13e4a3b143684bbc66446c7"
server_key_iv: "60cc59ad7a19d5f7c6bc147671f7adf4"
server_key_version: "encryptionkey@0"
I20260620 00:10:38.258405 17373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "ecd964f334a746e2b2428661db02baa8"
format_stamp: "Formatted at 2026-06-20 00:10:38 on dist-test-slave-grqt"
server_key: "2ec40870b13e4a3b143684bbc66446c7"
server_key_iv: "60cc59ad7a19d5f7c6bc147671f7adf4"
server_key_version: "encryptionkey@0"
I20260620 00:10:38.259686 17373 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:10:38.260425 17388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:38.260623 17373 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:10:38.260708 17373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "ecd964f334a746e2b2428661db02baa8"
format_stamp: "Formatted at 2026-06-20 00:10:38 on dist-test-slave-grqt"
server_key: "2ec40870b13e4a3b143684bbc66446c7"
server_key_iv: "60cc59ad7a19d5f7c6bc147671f7adf4"
server_key_version: "encryptionkey@0"
I20260620 00:10:38.260763 17373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:10:38.280548 17373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:10:38.282557 17373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:10:38.282668 17373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:10:38.286734 17373 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45859
I20260620 00:10:38.286787 17440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45859 every 8 connection(s)
I20260620 00:10:38.287276 17373 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:10:38.288017 17441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:10:38.290287 17441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecd964f334a746e2b2428661db02baa8: Bootstrap starting.
I20260620 00:10:38.290865 17441 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecd964f334a746e2b2428661db02baa8: Neither blocks nor log segments found. Creating new log.
I20260620 00:10:38.291162 17441 log.cc:824] T 00000000000000000000000000000000 P ecd964f334a746e2b2428661db02baa8: Log is configured to *not* fsync() on all Append() calls
I20260620 00:10:38.291829 17441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecd964f334a746e2b2428661db02baa8: No bootstrap required, opened a new log
I20260620 00:10:38.293469 17441 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecd964f334a746e2b2428661db02baa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd964f334a746e2b2428661db02baa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45859 } }
I20260620 00:10:38.293625 17441 raft_consensus.cc:374] T 00000000000000000000000000000000 P ecd964f334a746e2b2428661db02baa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:10:38.293648 17441 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecd964f334a746e2b2428661db02baa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecd964f334a746e2b2428661db02baa8, State: Initialized, Role: FOLLOWER
I20260620 00:10:38.293820 17441 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecd964f334a746e2b2428661db02baa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd964f334a746e2b2428661db02baa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45859 } }
I20260620 00:10:38.293906 17441 raft_consensus.cc:388] T 00000000000000000000000000000000 P ecd964f334a746e2b2428661db02baa8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:10:38.293970 17441 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecd964f334a746e2b2428661db02baa8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:10:38.294018 17441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecd964f334a746e2b2428661db02baa8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:10:38.295253 17441 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecd964f334a746e2b2428661db02baa8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd964f334a746e2b2428661db02baa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45859 } }
I20260620 00:10:38.295395 17441 leader_election.cc:302] T 00000000000000000000000000000000 P ecd964f334a746e2b2428661db02baa8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecd964f334a746e2b2428661db02baa8; no voters:
I20260620 00:10:38.295542 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 17373
I20260620 00:10:38.295576 17441 leader_election.cc:288] T 00000000000000000000000000000000 P ecd964f334a746e2b2428661db02baa8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:10:38.295647 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:10:38.295640 17444 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ecd964f334a746e2b2428661db02baa8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:10:38.295794 17444 raft_consensus.cc:686] T 00000000000000000000000000000000 P ecd964f334a746e2b2428661db02baa8 [term 1 LEADER]: Becoming Leader. State: Replica: ecd964f334a746e2b2428661db02baa8, State: Running, Role: LEADER
I20260620 00:10:38.295841 17441 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecd964f334a746e2b2428661db02baa8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:10:38.295863 392 external_mini_cluster.cc:1428] Setting key 04ee225a9b1460113e1cae91ec4e6ced
I20260620 00:10:38.295879 17444 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecd964f334a746e2b2428661db02baa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd964f334a746e2b2428661db02baa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45859 } }
I20260620 00:10:38.296303 17445 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecd964f334a746e2b2428661db02baa8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ecd964f334a746e2b2428661db02baa8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd964f334a746e2b2428661db02baa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45859 } } }
I20260620 00:10:38.296341 17446 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecd964f334a746e2b2428661db02baa8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecd964f334a746e2b2428661db02baa8. Latest consensus state: current_term: 1 leader_uuid: "ecd964f334a746e2b2428661db02baa8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd964f334a746e2b2428661db02baa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45859 } } }
I20260620 00:10:38.296382 17445 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecd964f334a746e2b2428661db02baa8 [sys.catalog]: This master's current role is: LEADER
I20260620 00:10:38.296384 17446 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecd964f334a746e2b2428661db02baa8 [sys.catalog]: This master's current role is: LEADER
I20260620 00:10:38.297151 17451 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:10:38.297713 17451 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:10:38.299292 17451 catalog_manager.cc:1347] Generated new cluster ID: cc6059c308734363968643714bcf9b4a
I20260620 00:10:38.299345 17451 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 00:10:38 dist-test-slave-grqt krb5kdc[17360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:10:38.306571 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:38.296513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55372 (local address 127.0.98.62:45859)
0620 00:10:38.296911 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:38.296921 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:38.296957 (+ 36us) server_negotiation.cc:407] Connection header received
0620 00:10:38.297088 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:38.297097 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:38.297163 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:38.297313 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:38.298241 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:38.299554 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:38.300301 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:38.300732 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:38.300869 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:38.303215 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:38.303224 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:38.303224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:10:38.303235 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:38.304909 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:38.305271 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:38.305274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:38.305274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:38.305418 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:38.305652 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:38.305653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:38.305653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:38.305889 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:38.306060 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:38.306157 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 00:10:38.306395 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":89,"threads_started":1}
I20260620 00:10:38.312654 17451 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:10:38.313431 17451 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:10:38.319319 17451 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ecd964f334a746e2b2428661db02baa8: Generated new TSK 0
I20260620 00:10:38.319504 17451 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:10:38.352931 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45859
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:10:38.436281 17467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:10:38.436488 17467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:10:38.436512 17467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:10:38.437690 17467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:10:38.437732 17467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:10:38.437759 17467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:10:38.439253 17467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:10:38.439453 17467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:10:38.439641 17467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:10:38.444517 17473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:38.444588 17475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:38.444556 17472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:38.444774 17467 server_base.cc:1034] running on GCE node
I20260620 00:10:38.445025 17467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:10:38.445235 17467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:10:38.446382 17467 hybrid_clock.cc:648] HybridClock initialized: now 1781914238446365 us; error 32 us; skew 500 ppm
Jun 20 00:10:38 dist-test-slave-grqt krb5kdc[17360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:10:38.448948 17467 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:10:38.449469 17467 webserver.cc:460] Webserver started at http://127.0.98.1:41845/ using document root <none> and password file <none>
I20260620 00:10:38.449671 17467 fs_manager.cc:359] Metadata directory not provided
I20260620 00:10:38.449712 17467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:10:38.449800 17467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:10:38.450809 17467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-0/data/instance:
uuid: "cfcf368755fb4e5aab083cd15c49c46a"
format_stamp: "Formatted at 2026-06-20 00:10:38 on dist-test-slave-grqt"
server_key: "fe4e735d63922da1c255b7b0ea8ddb2d"
server_key_iv: "90daf7a17ab5331380389ecb73f2f00b"
server_key_version: "encryptionkey@0"
I20260620 00:10:38.451121 17467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cfcf368755fb4e5aab083cd15c49c46a"
format_stamp: "Formatted at 2026-06-20 00:10:38 on dist-test-slave-grqt"
server_key: "fe4e735d63922da1c255b7b0ea8ddb2d"
server_key_iv: "90daf7a17ab5331380389ecb73f2f00b"
server_key_version: "encryptionkey@0"
I20260620 00:10:38.452383 17467 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:10:38.453152 17482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:38.453333 17467 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:10:38.453485 17467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-0/wal
uuid: "cfcf368755fb4e5aab083cd15c49c46a"
format_stamp: "Formatted at 2026-06-20 00:10:38 on dist-test-slave-grqt"
server_key: "fe4e735d63922da1c255b7b0ea8ddb2d"
server_key_iv: "90daf7a17ab5331380389ecb73f2f00b"
server_key_version: "encryptionkey@0"
I20260620 00:10:38.453570 17467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:10:38.471586 17467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:10:38.473953 17467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:10:38.474114 17467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:10:38.474346 17467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:10:38.474666 17467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:10:38.474721 17467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:38.474763 17467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:10:38.474807 17467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:38.480752 17467 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33855
I20260620 00:10:38.480798 17595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33855 every 8 connection(s)
I20260620 00:10:38.481105 17467 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 00:10:38 dist-test-slave-grqt krb5kdc[17360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:10:38.488287 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 17467
I20260620 00:10:38.488425 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:10:38.488624 392 external_mini_cluster.cc:1428] Setting key d464597749b8078be87f9d9ac0a7f107
I20260620 00:10:38.490702 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:38.481831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57261 (local address 127.0.98.62:45859)
0620 00:10:38.481906 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:38.481908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:38.482146 (+ 238us) server_negotiation.cc:407] Connection header received
0620 00:10:38.483074 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:38.483075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:38.483086 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:38.483150 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:38.483736 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:38.484251 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:38.485251 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:38.485537 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:38.485593 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:38.487615 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:38.487616 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:38.487617 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:10:38.487623 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:38.489273 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:38.489733 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:38.489734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:38.489735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:38.489794 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:38.490055 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:38.490056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:38.490057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:38.490172 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:38.490249 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:38.490528 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 00:10:38.490610 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 00:10:38.490908 17598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:38.481809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45859 (local address 127.0.98.1:57261)
0620 00:10:38.481995 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:38.482037 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:38.482944 (+ 907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:38.483210 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:38.483213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:38.483466 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:10:38.483654 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:38.483658 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:38.484336 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:38.484337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:38.485172 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:38.485174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:38.485616 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:38.485616 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:38.485713 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:38.486152 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:10:38.486160 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:10:38.487534 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:10:38.489398 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:10:38.489407 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:10:38.489409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:10:38.489639 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:10:38.489862 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:10:38.489863 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:10:38.489864 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:10:38.489984 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:10:38.490240 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:10:38.490247 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:10:38.490389 (+ 142us) client_negotiation.cc:769] Sending connection context
0620 00:10:38.490507 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 00:10:38.490667 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":98,"threads_started":1}
I20260620 00:10:38.491746 17596 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45859
I20260620 00:10:38.491834 17596 heartbeater.cc:464] Registering TS with master...
I20260620 00:10:38.492027 17596 heartbeater.cc:511] Master 127.0.98.62:45859 requested a full tablet report, sending...
I20260620 00:10:38.492596 17405 ts_manager.cc:194] Registered new tserver with Master: cfcf368755fb4e5aab083cd15c49c46a (127.0.98.1:33855)
I20260620 00:10:38.493235 17405 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57261
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:10:38.517711 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45859
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:10:38.598721 17601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:10:38.598922 17601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:10:38.598954 17601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:10:38.600035 17601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:10:38.600070 17601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:10:38.600095 17601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:10:38.601512 17601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:10:38.601696 17601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:10:38.601882 17601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:10:38.606913 17606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:38.606968 17601 server_base.cc:1034] running on GCE node
W20260620 00:10:38.606981 17607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:38.606913 17609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:38.607388 17601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:10:38.607600 17601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:10:38.608753 17601 hybrid_clock.cc:648] HybridClock initialized: now 1781914238608720 us; error 48 us; skew 500 ppm
Jun 20 00:10:38 dist-test-slave-grqt krb5kdc[17360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:10:38.611480 17601 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:10:38.612021 17601 webserver.cc:460] Webserver started at http://127.0.98.2:41871/ using document root <none> and password file <none>
I20260620 00:10:38.612290 17601 fs_manager.cc:359] Metadata directory not provided
I20260620 00:10:38.612363 17601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:10:38.612519 17601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:10:38.613662 17601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-1/data/instance:
uuid: "dc145b17a6aa40009175ef4101aabd7c"
format_stamp: "Formatted at 2026-06-20 00:10:38 on dist-test-slave-grqt"
server_key: "c632d9fc257fe816247b4b9fcaae466d"
server_key_iv: "c5876f91b85c3e7e3ecbb895856dccac"
server_key_version: "encryptionkey@0"
I20260620 00:10:38.614020 17601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-1/wal/instance:
uuid: "dc145b17a6aa40009175ef4101aabd7c"
format_stamp: "Formatted at 2026-06-20 00:10:38 on dist-test-slave-grqt"
server_key: "c632d9fc257fe816247b4b9fcaae466d"
server_key_iv: "c5876f91b85c3e7e3ecbb895856dccac"
server_key_version: "encryptionkey@0"
I20260620 00:10:38.615396 17601 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:10:38.616341 17616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:38.616562 17601 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:10:38.616679 17601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-1/wal
uuid: "dc145b17a6aa40009175ef4101aabd7c"
format_stamp: "Formatted at 2026-06-20 00:10:38 on dist-test-slave-grqt"
server_key: "c632d9fc257fe816247b4b9fcaae466d"
server_key_iv: "c5876f91b85c3e7e3ecbb895856dccac"
server_key_version: "encryptionkey@0"
I20260620 00:10:38.616765 17601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:10:38.630825 17601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:10:38.633148 17601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:10:38.633324 17601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:10:38.633639 17601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:10:38.633977 17601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:10:38.634030 17601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:38.634071 17601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:10:38.634101 17601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:38.640331 17601 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38053
I20260620 00:10:38.640506 17729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38053 every 8 connection(s)
I20260620 00:10:38.640798 17601 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:10:38.642678 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 17601
I20260620 00:10:38.642803 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:10:38.643040 392 external_mini_cluster.cc:1428] Setting key ec18f3d60f55c23c0e5161b5e0846c47
Jun 20 00:10:38 dist-test-slave-grqt krb5kdc[17360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 00:10:38.651523 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:38.641910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59373 (local address 127.0.98.62:45859)
0620 00:10:38.641958 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:38.641960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:38.642226 (+ 266us) server_negotiation.cc:407] Connection header received
0620 00:10:38.643276 (+ 1050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:38.643277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:38.643288 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:38.643372 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:38.644167 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:38.644956 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:38.646066 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:38.646279 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:38.646368 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:38.648603 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:38.648604 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:38.648604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:10:38.648612 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:38.650092 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:38.650535 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:38.650536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:38.650536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:38.650595 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:38.650860 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:38.650861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:38.650861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:38.650987 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:38.651068 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:38.651344 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 00:10:38.651428 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 00:10:38.651775 17732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:38.641888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45859 (local address 127.0.98.2:59373)
0620 00:10:38.642092 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:38.642128 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:38.643149 (+ 1021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:38.643467 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:38.643479 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:38.643853 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:10:38.644064 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:38.644068 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:38.645051 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:38.645051 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:38.645973 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:38.645976 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:38.646377 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:38.646378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:38.646525 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:38.647083 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:10:38.647091 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:10:38.648494 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:10:38.650198 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:10:38.650205 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:10:38.650207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:10:38.650437 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:10:38.650662 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:10:38.650663 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:10:38.650664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:10:38.650803 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:10:38.651058 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:10:38.651064 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:10:38.651209 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 00:10:38.651323 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 00:10:38.651493 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":131,"threads_started":1}
I20260620 00:10:38.652333 17730 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45859
I20260620 00:10:38.652437 17730 heartbeater.cc:464] Registering TS with master...
I20260620 00:10:38.652679 17730 heartbeater.cc:511] Master 127.0.98.62:45859 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 00:10:38.653174 17405 ts_manager.cc:194] Registered new tserver with Master: dc145b17a6aa40009175ef4101aabd7c (127.0.98.2:38053)
I20260620 00:10:38.653625 17405 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59373
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:10:38.669059 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45859
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:10:38.750656 17735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:10:38.750878 17735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:10:38.750902 17735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:10:38.752116 17735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:10:38.752162 17735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:10:38.752189 17735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:10:38.753894 17735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:10:38.754149 17735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:10:38.754390 17735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:10:38.759549 17741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:38.759552 17743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:38.759545 17740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:38.759766 17735 server_base.cc:1034] running on GCE node
I20260620 00:10:38.759977 17735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:10:38.760233 17735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:10:38.761464 17735 hybrid_clock.cc:648] HybridClock initialized: now 1781914238761436 us; error 59 us; skew 500 ppm
Jun 20 00:10:38 dist-test-slave-grqt krb5kdc[17360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:10:38.764196 17735 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:10:38.764631 17735 webserver.cc:460] Webserver started at http://127.0.98.3:41835/ using document root <none> and password file <none>
I20260620 00:10:38.764858 17735 fs_manager.cc:359] Metadata directory not provided
I20260620 00:10:38.764904 17735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:10:38.765048 17735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:10:38.766183 17735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-2/data/instance:
uuid: "2429c76f5b3e47d4a2255a5799c171e0"
format_stamp: "Formatted at 2026-06-20 00:10:38 on dist-test-slave-grqt"
server_key: "9c44f0c1c2119f9f20b3a3372fb4dc8f"
server_key_iv: "a3d8dec676e0fd37dca6f95bba735207"
server_key_version: "encryptionkey@0"
I20260620 00:10:38.766567 17735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2429c76f5b3e47d4a2255a5799c171e0"
format_stamp: "Formatted at 2026-06-20 00:10:38 on dist-test-slave-grqt"
server_key: "9c44f0c1c2119f9f20b3a3372fb4dc8f"
server_key_iv: "a3d8dec676e0fd37dca6f95bba735207"
server_key_version: "encryptionkey@0"
I20260620 00:10:38.767946 17735 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:10:38.768709 17750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:38.768935 17735 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:10:38.769024 17735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-2/wal
uuid: "2429c76f5b3e47d4a2255a5799c171e0"
format_stamp: "Formatted at 2026-06-20 00:10:38 on dist-test-slave-grqt"
server_key: "9c44f0c1c2119f9f20b3a3372fb4dc8f"
server_key_iv: "a3d8dec676e0fd37dca6f95bba735207"
server_key_version: "encryptionkey@0"
I20260620 00:10:38.769088 17735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:10:38.782871 17735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:10:38.785125 17735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:10:38.785247 17735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:10:38.785526 17735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:10:38.785861 17735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:10:38.785894 17735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:38.785918 17735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:10:38.785972 17735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:38.791744 17735 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38257
I20260620 00:10:38.791832 17863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38257 every 8 connection(s)
I20260620 00:10:38.792167 17735 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:10:38.794059 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 17735
I20260620 00:10:38.794202 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:10:38.794417 392 external_mini_cluster.cc:1428] Setting key b66edaebe83bb5b50a99891d059ef6a5
Jun 20 00:10:38 dist-test-slave-grqt krb5kdc[17360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:10:38.802883 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:38.792936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42869 (local address 127.0.98.62:45859)
0620 00:10:38.793044 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:38.793046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:38.793243 (+ 197us) server_negotiation.cc:407] Connection header received
0620 00:10:38.794518 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:38.794519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:38.794533 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:38.794611 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:38.795564 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:38.796082 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:38.797234 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:38.797520 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:38.797577 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:38.799749 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:38.799750 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:38.799750 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:10:38.799757 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:38.801469 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:38.801895 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:38.801896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:38.801897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:38.801978 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:38.802226 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:38.802227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:38.802228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:38.802361 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:38.802440 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:38.802729 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 00:10:38.802811 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 00:10:38.802951 17866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:38.792860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45859 (local address 127.0.98.3:42869)
0620 00:10:38.793080 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:38.793129 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:38.794389 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:38.794758 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:38.794763 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:38.795202 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:10:38.795493 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:38.795498 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:38.796218 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:38.796219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:38.797117 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:38.797119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:38.797600 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:38.797601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:38.797758 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:38.798330 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:10:38.798339 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:10:38.799667 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:10:38.801559 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:10:38.801565 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:10:38.801567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:10:38.801796 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:10:38.802044 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:10:38.802045 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:10:38.802045 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:10:38.802143 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:10:38.802454 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:10:38.802458 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:10:38.802577 (+ 119us) client_negotiation.cc:769] Sending connection context
0620 00:10:38.802666 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 00:10:38.802787 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":127,"threads_started":1}
I20260620 00:10:38.803509 17864 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45859
I20260620 00:10:38.803607 17864 heartbeater.cc:464] Registering TS with master...
I20260620 00:10:38.803751 17864 heartbeater.cc:511] Master 127.0.98.62:45859 requested a full tablet report, sending...
I20260620 00:10:38.804167 17405 ts_manager.cc:194] Registered new tserver with Master: 2429c76f5b3e47d4a2255a5799c171e0 (127.0.98.3:38257)
I20260620 00:10:38.804522 17405 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42869
I20260620 00:10:38.806186 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:10:38.813115 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:38.807464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55378 (local address 127.0.98.62:45859)
0620 00:10:38.807535 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:38.807536 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:38.807603 (+ 67us) server_negotiation.cc:407] Connection header received
0620 00:10:38.807638 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:38.807638 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:38.807647 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:38.807688 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:38.808219 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:38.808671 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:38.809513 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:38.809691 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:38.809744 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:38.810944 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:38.810945 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:38.810945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:10:38.810950 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:38.812112 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:38.812434 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:38.812434 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:38.812435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:38.812485 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:38.812637 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:38.812637 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:38.812637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:38.812761 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:38.812828 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:38.812918 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 00:10:38.812999 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 00:10:38.814919 17405 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55378:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:10:38.816485 17405 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:10:38.823565 17877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:38.819361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47396 (local address 127.0.98.1:33855)
0620 00:10:38.819572 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:38.819576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:38.819586 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:10:38.819623 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:38.819659 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:38.819678 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:38.819865 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:10:38.819997 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:38.821038 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:38.822445 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:38.823192 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:38.823272 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:38.823331 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:38.823463 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 00:10:38.823497 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":103,"threads_started":1}
I20260620 00:10:38.823580 17875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:38.819160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33855 (local address 127.0.0.1:47396)
0620 00:10:38.819345 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:38.819355 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:38.819397 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:38.819890 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:38.819891 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:38.819893 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:10:38.819927 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:38.819928 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:38.821100 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:38.821100 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:38.822355 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:38.822357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:38.823274 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:38.823275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:38.823352 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:38.823367 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 00:10:38.823443 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 00:10:38.823498 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":6,"thread_start_us":40,"threads_started":1}
I20260620 00:10:38.823915 17872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:38.818712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38053 (local address 127.0.0.1:54376)
0620 00:10:38.818867 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:38.818899 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:38.818982 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:38.819405 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:38.819408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:38.819410 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:10:38.819486 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:38.819500 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:38.820926 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:38.820927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:38.822351 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:38.822354 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:38.823686 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:38.823686 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:38.823744 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:38.823747 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:10:38.823798 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 00:10:38.823837 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":81,"thread_start_us":84,"threads_started":1}
I20260620 00:10:38.824147 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:38.818835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54376 (local address 127.0.98.2:38053)
0620 00:10:38.819026 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:38.819031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:38.819042 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:10:38.819081 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:38.819083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:38.819099 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:38.819584 (+ 485us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:10:38.819682 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:38.820779 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:38.822444 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:38.823192 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:38.823272 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:38.823330 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:38.824020 (+ 690us) server_negotiation.cc:299] Negotiation successful
0620 00:10:38.824065 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":101,"threads_started":1}
I20260620 00:10:38.824204 17874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:38.818991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38257 (local address 127.0.0.1:32796)
0620 00:10:38.819094 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:38.819104 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:38.819153 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:38.819700 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:38.819701 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:38.819704 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:10:38.819765 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:38.819767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:38.820854 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:38.820855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:38.823082 (+ 2227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:38.823084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:38.824005 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:38.824006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:38.824064 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:38.824068 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:10:38.824099 (+ 31us) client_negotiation.cc:240] Negotiation successful
0620 00:10:38.824132 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":1417,"thread_start_us":69,"threads_started":1}
I20260620 00:10:38.824472 17876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:38.819192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32796 (local address 127.0.98.3:38257)
0620 00:10:38.819440 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:38.819446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:38.819457 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:10:38.819574 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:38.819577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:38.819595 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:38.819837 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:10:38.819919 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:38.820752 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:38.823178 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:38.823907 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:38.824291 (+ 384us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:38.824355 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:38.824383 (+ 28us) server_negotiation.cc:299] Negotiation successful
0620 00:10:38.824414 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":101,"threads_started":1}
I20260620 00:10:38.824542 17530 tablet_service.cc:1467] Processing CreateTablet for tablet 3dcce3cf90c84ebba6be6b2f82266b50 (DEFAULT_TABLE table=test-table [id=70cb399a80ac4bacbda5af1134efbd8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:10:38.824923 17530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dcce3cf90c84ebba6be6b2f82266b50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:10:38.826072 17798 tablet_service.cc:1467] Processing CreateTablet for tablet 3dcce3cf90c84ebba6be6b2f82266b50 (DEFAULT_TABLE table=test-table [id=70cb399a80ac4bacbda5af1134efbd8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:10:38.826154 17664 tablet_service.cc:1467] Processing CreateTablet for tablet 3dcce3cf90c84ebba6be6b2f82266b50 (DEFAULT_TABLE table=test-table [id=70cb399a80ac4bacbda5af1134efbd8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:10:38.826320 17798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dcce3cf90c84ebba6be6b2f82266b50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:10:38.826471 17664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dcce3cf90c84ebba6be6b2f82266b50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:10:38.828343 17878 tablet_bootstrap.cc:492] T 3dcce3cf90c84ebba6be6b2f82266b50 P cfcf368755fb4e5aab083cd15c49c46a: Bootstrap starting.
I20260620 00:10:38.829028 17878 tablet_bootstrap.cc:654] T 3dcce3cf90c84ebba6be6b2f82266b50 P cfcf368755fb4e5aab083cd15c49c46a: Neither blocks nor log segments found. Creating new log.
I20260620 00:10:38.829526 17878 log.cc:824] T 3dcce3cf90c84ebba6be6b2f82266b50 P cfcf368755fb4e5aab083cd15c49c46a: Log is configured to *not* fsync() on all Append() calls
I20260620 00:10:38.829599 17879 tablet_bootstrap.cc:492] T 3dcce3cf90c84ebba6be6b2f82266b50 P 2429c76f5b3e47d4a2255a5799c171e0: Bootstrap starting.
I20260620 00:10:38.829835 17880 tablet_bootstrap.cc:492] T 3dcce3cf90c84ebba6be6b2f82266b50 P dc145b17a6aa40009175ef4101aabd7c: Bootstrap starting.
I20260620 00:10:38.830232 17879 tablet_bootstrap.cc:654] T 3dcce3cf90c84ebba6be6b2f82266b50 P 2429c76f5b3e47d4a2255a5799c171e0: Neither blocks nor log segments found. Creating new log.
I20260620 00:10:38.830416 17880 tablet_bootstrap.cc:654] T 3dcce3cf90c84ebba6be6b2f82266b50 P dc145b17a6aa40009175ef4101aabd7c: Neither blocks nor log segments found. Creating new log.
I20260620 00:10:38.830775 17879 log.cc:824] T 3dcce3cf90c84ebba6be6b2f82266b50 P 2429c76f5b3e47d4a2255a5799c171e0: Log is configured to *not* fsync() on all Append() calls
I20260620 00:10:38.830935 17880 log.cc:824] T 3dcce3cf90c84ebba6be6b2f82266b50 P dc145b17a6aa40009175ef4101aabd7c: Log is configured to *not* fsync() on all Append() calls
I20260620 00:10:38.831647 17878 tablet_bootstrap.cc:492] T 3dcce3cf90c84ebba6be6b2f82266b50 P cfcf368755fb4e5aab083cd15c49c46a: No bootstrap required, opened a new log
I20260620 00:10:38.831658 17879 tablet_bootstrap.cc:492] T 3dcce3cf90c84ebba6be6b2f82266b50 P 2429c76f5b3e47d4a2255a5799c171e0: No bootstrap required, opened a new log
I20260620 00:10:38.831738 17878 ts_tablet_manager.cc:1397] T 3dcce3cf90c84ebba6be6b2f82266b50 P cfcf368755fb4e5aab083cd15c49c46a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 00:10:38.831753 17879 ts_tablet_manager.cc:1397] T 3dcce3cf90c84ebba6be6b2f82266b50 P 2429c76f5b3e47d4a2255a5799c171e0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:10:38.833123 17878 raft_consensus.cc:348] T 3dcce3cf90c84ebba6be6b2f82266b50 P cfcf368755fb4e5aab083cd15c49c46a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc145b17a6aa40009175ef4101aabd7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38053 } } peers { permanent_uuid: "2429c76f5b3e47d4a2255a5799c171e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38257 } } peers { permanent_uuid: "cfcf368755fb4e5aab083cd15c49c46a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33855 } }
I20260620 00:10:38.833122 17879 raft_consensus.cc:348] T 3dcce3cf90c84ebba6be6b2f82266b50 P 2429c76f5b3e47d4a2255a5799c171e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc145b17a6aa40009175ef4101aabd7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38053 } } peers { permanent_uuid: "2429c76f5b3e47d4a2255a5799c171e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38257 } } peers { permanent_uuid: "cfcf368755fb4e5aab083cd15c49c46a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33855 } }
I20260620 00:10:38.833299 17879 raft_consensus.cc:374] T 3dcce3cf90c84ebba6be6b2f82266b50 P 2429c76f5b3e47d4a2255a5799c171e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:10:38.833302 17878 raft_consensus.cc:374] T 3dcce3cf90c84ebba6be6b2f82266b50 P cfcf368755fb4e5aab083cd15c49c46a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:10:38.833351 17879 raft_consensus.cc:729] T 3dcce3cf90c84ebba6be6b2f82266b50 P 2429c76f5b3e47d4a2255a5799c171e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2429c76f5b3e47d4a2255a5799c171e0, State: Initialized, Role: FOLLOWER
I20260620 00:10:38.833353 17878 raft_consensus.cc:729] T 3dcce3cf90c84ebba6be6b2f82266b50 P cfcf368755fb4e5aab083cd15c49c46a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfcf368755fb4e5aab083cd15c49c46a, State: Initialized, Role: FOLLOWER
I20260620 00:10:38.833534 17879 consensus_queue.cc:260] T 3dcce3cf90c84ebba6be6b2f82266b50 P 2429c76f5b3e47d4a2255a5799c171e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc145b17a6aa40009175ef4101aabd7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38053 } } peers { permanent_uuid: "2429c76f5b3e47d4a2255a5799c171e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38257 } } peers { permanent_uuid: "cfcf368755fb4e5aab083cd15c49c46a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33855 } }
I20260620 00:10:38.833521 17878 consensus_queue.cc:260] T 3dcce3cf90c84ebba6be6b2f82266b50 P cfcf368755fb4e5aab083cd15c49c46a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc145b17a6aa40009175ef4101aabd7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38053 } } peers { permanent_uuid: "2429c76f5b3e47d4a2255a5799c171e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38257 } } peers { permanent_uuid: "cfcf368755fb4e5aab083cd15c49c46a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33855 } }
I20260620 00:10:38.833823 17879 ts_tablet_manager.cc:1428] T 3dcce3cf90c84ebba6be6b2f82266b50 P 2429c76f5b3e47d4a2255a5799c171e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:10:38.833868 17864 heartbeater.cc:503] Master 127.0.98.62:45859 was elected leader, sending a full tablet report...
I20260620 00:10:38.833969 17596 heartbeater.cc:503] Master 127.0.98.62:45859 was elected leader, sending a full tablet report...
I20260620 00:10:38.834170 17878 ts_tablet_manager.cc:1428] T 3dcce3cf90c84ebba6be6b2f82266b50 P cfcf368755fb4e5aab083cd15c49c46a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 00:10:38.834924 17880 tablet_bootstrap.cc:492] T 3dcce3cf90c84ebba6be6b2f82266b50 P dc145b17a6aa40009175ef4101aabd7c: No bootstrap required, opened a new log
I20260620 00:10:38.835045 17880 ts_tablet_manager.cc:1397] T 3dcce3cf90c84ebba6be6b2f82266b50 P dc145b17a6aa40009175ef4101aabd7c: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260620 00:10:38.836532 17880 raft_consensus.cc:348] T 3dcce3cf90c84ebba6be6b2f82266b50 P dc145b17a6aa40009175ef4101aabd7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc145b17a6aa40009175ef4101aabd7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38053 } } peers { permanent_uuid: "2429c76f5b3e47d4a2255a5799c171e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38257 } } peers { permanent_uuid: "cfcf368755fb4e5aab083cd15c49c46a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33855 } }
I20260620 00:10:38.836678 17880 raft_consensus.cc:374] T 3dcce3cf90c84ebba6be6b2f82266b50 P dc145b17a6aa40009175ef4101aabd7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:10:38.836700 17880 raft_consensus.cc:729] T 3dcce3cf90c84ebba6be6b2f82266b50 P dc145b17a6aa40009175ef4101aabd7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc145b17a6aa40009175ef4101aabd7c, State: Initialized, Role: FOLLOWER
I20260620 00:10:38.836840 17880 consensus_queue.cc:260] T 3dcce3cf90c84ebba6be6b2f82266b50 P dc145b17a6aa40009175ef4101aabd7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc145b17a6aa40009175ef4101aabd7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38053 } } peers { permanent_uuid: "2429c76f5b3e47d4a2255a5799c171e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38257 } } peers { permanent_uuid: "cfcf368755fb4e5aab083cd15c49c46a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33855 } }
I20260620 00:10:38.837152 17880 ts_tablet_manager.cc:1428] T 3dcce3cf90c84ebba6be6b2f82266b50 P dc145b17a6aa40009175ef4101aabd7c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:10:38.837173 17730 heartbeater.cc:503] Master 127.0.98.62:45859 was elected leader, sending a full tablet report...
I20260620 00:10:38.881093 17885 raft_consensus.cc:482] T 3dcce3cf90c84ebba6be6b2f82266b50 P 2429c76f5b3e47d4a2255a5799c171e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:10:38.881227 17885 raft_consensus.cc:504] T 3dcce3cf90c84ebba6be6b2f82266b50 P 2429c76f5b3e47d4a2255a5799c171e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc145b17a6aa40009175ef4101aabd7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38053 } } peers { permanent_uuid: "2429c76f5b3e47d4a2255a5799c171e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38257 } } peers { permanent_uuid: "cfcf368755fb4e5aab083cd15c49c46a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33855 } }
I20260620 00:10:38.881620 17885 leader_election.cc:288] T 3dcce3cf90c84ebba6be6b2f82266b50 P 2429c76f5b3e47d4a2255a5799c171e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc145b17a6aa40009175ef4101aabd7c (127.0.98.2:38053), cfcf368755fb4e5aab083cd15c49c46a (127.0.98.1:33855)
I20260620 00:10:38.884902 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:38.881962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55423 (local address 127.0.98.2:38053)
0620 00:10:38.882037 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:38.882039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:38.882049 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:10:38.882079 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:38.882080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:38.882091 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:38.882151 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:10:38.882357 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:38.882937 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:38.884136 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:38.884524 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:38.884609 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:38.884620 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:38.884787 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 00:10:38.884827 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 00:10:38.884922 17866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:38.881829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38053 (local address 127.0.98.3:55423)
0620 00:10:38.881912 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:38.881929 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:38.881997 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:38.882191 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:38.882193 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:38.882194 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:10:38.882266 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:38.882268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:38.883024 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:38.883025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:38.884054 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:38.884069 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:38.884608 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:38.884609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:38.884693 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:38.884697 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:10:38.884768 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 00:10:38.884810 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":2}
I20260620 00:10:38.884953 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:38.882012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33855 (local address 127.0.98.3:36653)
0620 00:10:38.882194 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:38.882204 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:38.882254 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:38.882413 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:38.882413 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:38.882415 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:10:38.882483 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:38.882484 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:38.883115 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:38.883116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:38.884070 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:38.884074 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:38.884658 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:38.884659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:38.884746 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:38.884750 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:10:38.884812 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 00:10:38.884854 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":25,"thread_start_us":116,"threads_started":1}
I20260620 00:10:38.884943 17877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:38.882107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36653 (local address 127.0.98.1:33855)
0620 00:10:38.882160 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:38.882162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:38.882245 (+ 83us) server_negotiation.cc:407] Connection header received
0620 00:10:38.882303 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:38.882304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:38.882319 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:38.882395 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:10:38.882554 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:38.883029 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:38.884185 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:38.884560 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:38.884644 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:38.884655 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:38.884834 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 00:10:38.884884 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 00:10:38.885262 17684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcce3cf90c84ebba6be6b2f82266b50" candidate_uuid: "2429c76f5b3e47d4a2255a5799c171e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc145b17a6aa40009175ef4101aabd7c" is_pre_election: true
I20260620 00:10:38.885262 17550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcce3cf90c84ebba6be6b2f82266b50" candidate_uuid: "2429c76f5b3e47d4a2255a5799c171e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfcf368755fb4e5aab083cd15c49c46a" is_pre_election: true
I20260620 00:10:38.885449 17550 raft_consensus.cc:2454] T 3dcce3cf90c84ebba6be6b2f82266b50 P cfcf368755fb4e5aab083cd15c49c46a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2429c76f5b3e47d4a2255a5799c171e0 in term 0.
I20260620 00:10:38.885449 17684 raft_consensus.cc:2454] T 3dcce3cf90c84ebba6be6b2f82266b50 P dc145b17a6aa40009175ef4101aabd7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2429c76f5b3e47d4a2255a5799c171e0 in term 0.
I20260620 00:10:38.885690 17753 leader_election.cc:302] T 3dcce3cf90c84ebba6be6b2f82266b50 P 2429c76f5b3e47d4a2255a5799c171e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2429c76f5b3e47d4a2255a5799c171e0, dc145b17a6aa40009175ef4101aabd7c; no voters:
I20260620 00:10:38.885883 17885 raft_consensus.cc:2781] T 3dcce3cf90c84ebba6be6b2f82266b50 P 2429c76f5b3e47d4a2255a5799c171e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:10:38.885979 17885 raft_consensus.cc:482] T 3dcce3cf90c84ebba6be6b2f82266b50 P 2429c76f5b3e47d4a2255a5799c171e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:10:38.886027 17885 raft_consensus.cc:3037] T 3dcce3cf90c84ebba6be6b2f82266b50 P 2429c76f5b3e47d4a2255a5799c171e0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:10:38.886771 17885 raft_consensus.cc:504] T 3dcce3cf90c84ebba6be6b2f82266b50 P 2429c76f5b3e47d4a2255a5799c171e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc145b17a6aa40009175ef4101aabd7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38053 } } peers { permanent_uuid: "2429c76f5b3e47d4a2255a5799c171e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38257 } } peers { permanent_uuid: "cfcf368755fb4e5aab083cd15c49c46a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33855 } }
I20260620 00:10:38.886886 17885 leader_election.cc:288] T 3dcce3cf90c84ebba6be6b2f82266b50 P 2429c76f5b3e47d4a2255a5799c171e0 [CANDIDATE]: Term 1 election: Requested vote from peers dc145b17a6aa40009175ef4101aabd7c (127.0.98.2:38053), cfcf368755fb4e5aab083cd15c49c46a (127.0.98.1:33855)
I20260620 00:10:38.887127 17550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcce3cf90c84ebba6be6b2f82266b50" candidate_uuid: "2429c76f5b3e47d4a2255a5799c171e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfcf368755fb4e5aab083cd15c49c46a"
I20260620 00:10:38.887213 17550 raft_consensus.cc:3037] T 3dcce3cf90c84ebba6be6b2f82266b50 P cfcf368755fb4e5aab083cd15c49c46a [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:10:38.887916 17550 raft_consensus.cc:2454] T 3dcce3cf90c84ebba6be6b2f82266b50 P cfcf368755fb4e5aab083cd15c49c46a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2429c76f5b3e47d4a2255a5799c171e0 in term 1.
I20260620 00:10:38.888025 17684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcce3cf90c84ebba6be6b2f82266b50" candidate_uuid: "2429c76f5b3e47d4a2255a5799c171e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc145b17a6aa40009175ef4101aabd7c"
I20260620 00:10:38.888098 17753 leader_election.cc:302] T 3dcce3cf90c84ebba6be6b2f82266b50 P 2429c76f5b3e47d4a2255a5799c171e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2429c76f5b3e47d4a2255a5799c171e0, cfcf368755fb4e5aab083cd15c49c46a; no voters:
I20260620 00:10:38.888139 17684 raft_consensus.cc:3037] T 3dcce3cf90c84ebba6be6b2f82266b50 P dc145b17a6aa40009175ef4101aabd7c [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:10:38.888221 17885 raft_consensus.cc:2781] T 3dcce3cf90c84ebba6be6b2f82266b50 P 2429c76f5b3e47d4a2255a5799c171e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:10:38.888396 17885 raft_consensus.cc:686] T 3dcce3cf90c84ebba6be6b2f82266b50 P 2429c76f5b3e47d4a2255a5799c171e0 [term 1 LEADER]: Becoming Leader. State: Replica: 2429c76f5b3e47d4a2255a5799c171e0, State: Running, Role: LEADER
I20260620 00:10:38.888513 17885 consensus_queue.cc:237] T 3dcce3cf90c84ebba6be6b2f82266b50 P 2429c76f5b3e47d4a2255a5799c171e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc145b17a6aa40009175ef4101aabd7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38053 } } peers { permanent_uuid: "2429c76f5b3e47d4a2255a5799c171e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38257 } } peers { permanent_uuid: "cfcf368755fb4e5aab083cd15c49c46a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33855 } }
I20260620 00:10:38.889138 17684 raft_consensus.cc:2454] T 3dcce3cf90c84ebba6be6b2f82266b50 P dc145b17a6aa40009175ef4101aabd7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2429c76f5b3e47d4a2255a5799c171e0 in term 1.
I20260620 00:10:38.889519 17404 catalog_manager.cc:5495] T 3dcce3cf90c84ebba6be6b2f82266b50 P 2429c76f5b3e47d4a2255a5799c171e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2429c76f5b3e47d4a2255a5799c171e0 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "2429c76f5b3e47d4a2255a5799c171e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc145b17a6aa40009175ef4101aabd7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2429c76f5b3e47d4a2255a5799c171e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfcf368755fb4e5aab083cd15c49c46a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33855 } health_report { overall_health: UNKNOWN } } }
W20260620 00:10:38.891587 17731 tablet.cc:2369] T 3dcce3cf90c84ebba6be6b2f82266b50 P dc145b17a6aa40009175ef4101aabd7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:10:38.903846 17876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:38.901129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32810 (local address 127.0.98.3:38257)
0620 00:10:38.901178 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:38.901180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:38.901189 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:10:38.901249 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:38.901250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:38.901269 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:38.901367 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:10:38.901556 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:38.902087 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:38.902938 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:38.903176 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:38.903268 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:38.903474 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:10:38.903590 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:10:38.903693 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:38.903714 (+ 21us) server_negotiation.cc:299] Negotiation successful
0620 00:10:38.903735 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 00:10:38.905782 17684 raft_consensus.cc:1261] T 3dcce3cf90c84ebba6be6b2f82266b50 P dc145b17a6aa40009175ef4101aabd7c [term 1 FOLLOWER]: Refusing update from remote peer 2429c76f5b3e47d4a2255a5799c171e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:10:38.905782 17550 raft_consensus.cc:1261] T 3dcce3cf90c84ebba6be6b2f82266b50 P cfcf368755fb4e5aab083cd15c49c46a [term 1 FOLLOWER]: Refusing update from remote peer 2429c76f5b3e47d4a2255a5799c171e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:10:38.906157 17885 consensus_queue.cc:1035] T 3dcce3cf90c84ebba6be6b2f82266b50 P 2429c76f5b3e47d4a2255a5799c171e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcf368755fb4e5aab083cd15c49c46a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:10:38.906340 17888 consensus_queue.cc:1035] T 3dcce3cf90c84ebba6be6b2f82266b50 P 2429c76f5b3e47d4a2255a5799c171e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc145b17a6aa40009175ef4101aabd7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:10:38.908949 17891 mvcc.cc:205] Tried to move back new op lower bound from 7298720722554982400 to 7298720722488020992. Current Snapshot: MvccSnapshot[applied={T|T < 7298720722554982400}]
I20260620 00:10:38.910933 17893 mvcc.cc:205] Tried to move back new op lower bound from 7298720722554982400 to 7298720722488020992. Current Snapshot: MvccSnapshot[applied={T|T < 7298720722554982400 or (T in {7298720722554982400})}]
I20260620 00:10:38.911212 17404 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:10:38.911314 17404 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:10:38.912127 17404 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ecd964f334a746e2b2428661db02baa8: Sending DeleteTablet for 3 replicas of tablet 3dcce3cf90c84ebba6be6b2f82266b50
I20260620 00:10:38.912472 17664 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dcce3cf90c84ebba6be6b2f82266b50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:10:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54376
I20260620 00:10:38.912497 17798 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dcce3cf90c84ebba6be6b2f82266b50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:10:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32796
I20260620 00:10:38.912595 17530 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dcce3cf90c84ebba6be6b2f82266b50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:10:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47396
I20260620 00:10:38.912705 17901 tablet_replica.cc:330] T 3dcce3cf90c84ebba6be6b2f82266b50 P dc145b17a6aa40009175ef4101aabd7c: stopping tablet replica
I20260620 00:10:38.912808 17901 raft_consensus.cc:2229] T 3dcce3cf90c84ebba6be6b2f82266b50 P dc145b17a6aa40009175ef4101aabd7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:10:38.912837 17903 tablet_replica.cc:330] T 3dcce3cf90c84ebba6be6b2f82266b50 P cfcf368755fb4e5aab083cd15c49c46a: stopping tablet replica
I20260620 00:10:38.912932 17901 raft_consensus.cc:2258] T 3dcce3cf90c84ebba6be6b2f82266b50 P dc145b17a6aa40009175ef4101aabd7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:10:38.913058 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 17467
I20260620 00:10:38.913071 17903 raft_consensus.cc:2229] T 3dcce3cf90c84ebba6be6b2f82266b50 P cfcf368755fb4e5aab083cd15c49c46a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:10:38.913836 17901 ts_tablet_manager.cc:1905] T 3dcce3cf90c84ebba6be6b2f82266b50 P dc145b17a6aa40009175ef4101aabd7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:10:38.915380 17901 ts_tablet_manager.cc:1918] T 3dcce3cf90c84ebba6be6b2f82266b50 P dc145b17a6aa40009175ef4101aabd7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:10:38.915483 17901 log.cc:1196] T 3dcce3cf90c84ebba6be6b2f82266b50 P dc145b17a6aa40009175ef4101aabd7c: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-1/wal/wals/3dcce3cf90c84ebba6be6b2f82266b50
I20260620 00:10:38.915777 17901 ts_tablet_manager.cc:1939] T 3dcce3cf90c84ebba6be6b2f82266b50 P dc145b17a6aa40009175ef4101aabd7c: Deleting consensus metadata
I20260620 00:10:38.916209 17391 catalog_manager.cc:4841] TS dc145b17a6aa40009175ef4101aabd7c (127.0.98.2:38053): tablet 3dcce3cf90c84ebba6be6b2f82266b50 (table test-table [id=70cb399a80ac4bacbda5af1134efbd8f]) successfully deleted
I20260620 00:10:38.917610 17902 tablet_replica.cc:330] T 3dcce3cf90c84ebba6be6b2f82266b50 P 2429c76f5b3e47d4a2255a5799c171e0: stopping tablet replica
I20260620 00:10:38.917740 17902 raft_consensus.cc:2229] T 3dcce3cf90c84ebba6be6b2f82266b50 P 2429c76f5b3e47d4a2255a5799c171e0 [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:10:38.917847 17902 raft_consensus.cc:2258] T 3dcce3cf90c84ebba6be6b2f82266b50 P 2429c76f5b3e47d4a2255a5799c171e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:10:38.918268 17902 ts_tablet_manager.cc:1905] T 3dcce3cf90c84ebba6be6b2f82266b50 P 2429c76f5b3e47d4a2255a5799c171e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:10:38.919806 17902 ts_tablet_manager.cc:1918] T 3dcce3cf90c84ebba6be6b2f82266b50 P 2429c76f5b3e47d4a2255a5799c171e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:10:38.919894 17902 log.cc:1196] T 3dcce3cf90c84ebba6be6b2f82266b50 P 2429c76f5b3e47d4a2255a5799c171e0: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914001714546-392-0/minicluster-data/ts-2/wal/wals/3dcce3cf90c84ebba6be6b2f82266b50
I20260620 00:10:38.920161 17902 ts_tablet_manager.cc:1939] T 3dcce3cf90c84ebba6be6b2f82266b50 P 2429c76f5b3e47d4a2255a5799c171e0: Deleting consensus metadata
I20260620 00:10:38.920607 17391 catalog_manager.cc:4841] TS 2429c76f5b3e47d4a2255a5799c171e0 (127.0.98.3:38257): tablet 3dcce3cf90c84ebba6be6b2f82266b50 (table test-table [id=70cb399a80ac4bacbda5af1134efbd8f]) successfully deleted
W20260620 00:10:38.920781 17391 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33855 (error 108)
I20260620 00:10:38.920858 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 17601
I20260620 00:10:38.921397 17874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:38.921235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33855 (local address 127.0.0.1:47402)
0620 00:10:38.921288 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:38.921337 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 00:10:38.921728 17391 catalog_manager.cc:4584] TS cfcf368755fb4e5aab083cd15c49c46a (127.0.98.1:33855): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3dcce3cf90c84ebba6be6b2f82266b50: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33855: connect: Connection refused (error 111)
I20260620 00:10:38.928484 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 17735
I20260620 00:10:38.933583 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 17373
2026-06-20T00:10:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1960 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T00:10:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:10:38Z Disabled control of system clock
I20260620 00:10:38.961931 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35617
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 00:10:39.041751 17908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:10:39.041982 17908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:10:39.042006 17908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:10:39.043076 17908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:10:39.043108 17908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:10:39.043121 17908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:10:39.043133 17908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:10:39.043146 17908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:10:39.044399 17908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35617
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:10:39.044590 17908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:10:39.044816 17908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:10:39.049844 17914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:39.049850 17913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:39.049911 17916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:39.050207 17908 server_base.cc:1034] running on GCE node
I20260620 00:10:39.050459 17908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:10:39.050750 17908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:10:39.051913 17908 hybrid_clock.cc:648] HybridClock initialized: now 1781914239051890 us; error 54 us; skew 500 ppm
I20260620 00:10:39.053236 17908 webserver.cc:460] Webserver started at http://127.0.98.62:39433/ using document root <none> and password file <none>
I20260620 00:10:39.053522 17908 fs_manager.cc:359] Metadata directory not provided
I20260620 00:10:39.053592 17908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:10:39.053742 17908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:10:39.054814 17908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "e44f622f0b1e4af583ab7e70c159f1f1"
format_stamp: "Formatted at 2026-06-20 00:10:39 on dist-test-slave-grqt"
server_key: "6e644078ac3faf41acadb5d73e4c280a"
server_key_iv: "ef78ebfaa38d5c27b59bf1f51980040a"
server_key_version: "encryptionkey@0"
I20260620 00:10:39.055188 17908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "e44f622f0b1e4af583ab7e70c159f1f1"
format_stamp: "Formatted at 2026-06-20 00:10:39 on dist-test-slave-grqt"
server_key: "6e644078ac3faf41acadb5d73e4c280a"
server_key_iv: "ef78ebfaa38d5c27b59bf1f51980040a"
server_key_version: "encryptionkey@0"
I20260620 00:10:39.056468 17908 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:10:39.057161 17922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:39.057343 17908 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:10:39.057546 17908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "e44f622f0b1e4af583ab7e70c159f1f1"
format_stamp: "Formatted at 2026-06-20 00:10:39 on dist-test-slave-grqt"
server_key: "6e644078ac3faf41acadb5d73e4c280a"
server_key_iv: "ef78ebfaa38d5c27b59bf1f51980040a"
server_key_version: "encryptionkey@0"
I20260620 00:10:39.057618 17908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:10:39.065336 17908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:10:39.065672 17908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:10:39.065815 17908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:10:39.069751 17908 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35617
I20260620 00:10:39.069813 17974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35617 every 8 connection(s)
I20260620 00:10:39.070135 17908 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:10:39.071066 17975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:10:39.073222 17975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e44f622f0b1e4af583ab7e70c159f1f1: Bootstrap starting.
I20260620 00:10:39.073813 17975 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e44f622f0b1e4af583ab7e70c159f1f1: Neither blocks nor log segments found. Creating new log.
I20260620 00:10:39.074049 17975 log.cc:824] T 00000000000000000000000000000000 P e44f622f0b1e4af583ab7e70c159f1f1: Log is configured to *not* fsync() on all Append() calls
I20260620 00:10:39.074635 17975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e44f622f0b1e4af583ab7e70c159f1f1: No bootstrap required, opened a new log
I20260620 00:10:39.076108 17975 raft_consensus.cc:348] T 00000000000000000000000000000000 P e44f622f0b1e4af583ab7e70c159f1f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e44f622f0b1e4af583ab7e70c159f1f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35617 } }
I20260620 00:10:39.076232 17975 raft_consensus.cc:374] T 00000000000000000000000000000000 P e44f622f0b1e4af583ab7e70c159f1f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:10:39.076253 17975 raft_consensus.cc:729] T 00000000000000000000000000000000 P e44f622f0b1e4af583ab7e70c159f1f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e44f622f0b1e4af583ab7e70c159f1f1, State: Initialized, Role: FOLLOWER
I20260620 00:10:39.076356 17975 consensus_queue.cc:260] T 00000000000000000000000000000000 P e44f622f0b1e4af583ab7e70c159f1f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e44f622f0b1e4af583ab7e70c159f1f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35617 } }
I20260620 00:10:39.076400 17975 raft_consensus.cc:388] T 00000000000000000000000000000000 P e44f622f0b1e4af583ab7e70c159f1f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:10:39.076419 17975 raft_consensus.cc:482] T 00000000000000000000000000000000 P e44f622f0b1e4af583ab7e70c159f1f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:10:39.076445 17975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e44f622f0b1e4af583ab7e70c159f1f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:10:39.076428 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 17908
I20260620 00:10:39.076530 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:10:39.076732 392 external_mini_cluster.cc:1428] Setting key 444e6a528615856b86879ffd14660220
I20260620 00:10:39.077064 17975 raft_consensus.cc:504] T 00000000000000000000000000000000 P e44f622f0b1e4af583ab7e70c159f1f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e44f622f0b1e4af583ab7e70c159f1f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35617 } }
I20260620 00:10:39.077172 17975 leader_election.cc:302] T 00000000000000000000000000000000 P e44f622f0b1e4af583ab7e70c159f1f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e44f622f0b1e4af583ab7e70c159f1f1; no voters:
I20260620 00:10:39.077416 17975 leader_election.cc:288] T 00000000000000000000000000000000 P e44f622f0b1e4af583ab7e70c159f1f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:10:39.077463 17979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e44f622f0b1e4af583ab7e70c159f1f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:10:39.077606 17979 raft_consensus.cc:686] T 00000000000000000000000000000000 P e44f622f0b1e4af583ab7e70c159f1f1 [term 1 LEADER]: Becoming Leader. State: Replica: e44f622f0b1e4af583ab7e70c159f1f1, State: Running, Role: LEADER
I20260620 00:10:39.077687 17975 sys_catalog.cc:564] T 00000000000000000000000000000000 P e44f622f0b1e4af583ab7e70c159f1f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:10:39.077788 17979 consensus_queue.cc:237] T 00000000000000000000000000000000 P e44f622f0b1e4af583ab7e70c159f1f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e44f622f0b1e4af583ab7e70c159f1f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35617 } }
I20260620 00:10:39.078337 17982 sys_catalog.cc:455] T 00000000000000000000000000000000 P e44f622f0b1e4af583ab7e70c159f1f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e44f622f0b1e4af583ab7e70c159f1f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e44f622f0b1e4af583ab7e70c159f1f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35617 } } }
I20260620 00:10:39.078509 17982 sys_catalog.cc:458] T 00000000000000000000000000000000 P e44f622f0b1e4af583ab7e70c159f1f1 [sys.catalog]: This master's current role is: LEADER
I20260620 00:10:39.078776 17990 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:10:39.079273 17990 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:10:39.079278 17981 sys_catalog.cc:455] T 00000000000000000000000000000000 P e44f622f0b1e4af583ab7e70c159f1f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e44f622f0b1e4af583ab7e70c159f1f1. Latest consensus state: current_term: 1 leader_uuid: "e44f622f0b1e4af583ab7e70c159f1f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e44f622f0b1e4af583ab7e70c159f1f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35617 } } }
I20260620 00:10:39.079641 17981 sys_catalog.cc:458] T 00000000000000000000000000000000 P e44f622f0b1e4af583ab7e70c159f1f1 [sys.catalog]: This master's current role is: LEADER
I20260620 00:10:39.080927 17990 catalog_manager.cc:1347] Generated new cluster ID: a1a0b78fe7dc4244962a7bf88426a5a5
I20260620 00:10:39.080983 17990 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:10:39.082800 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:39.077234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47182 (local address 127.0.98.62:35617)
0620 00:10:39.077567 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:39.077577 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:39.077610 (+ 33us) server_negotiation.cc:407] Connection header received
0620 00:10:39.077728 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:39.077735 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:39.077826 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:39.077961 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:39.078204 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:39.079659 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:39.080626 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:39.081212 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:39.081341 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:39.082081 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:39.082085 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:39.082086 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:10:39.082294 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:39.082337 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:10:39.082350 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:39.082426 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:39.082495 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 00:10:39.082575 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":253,"thread_start_us":130,"threads_started":1}
I20260620 00:10:39.088106 17990 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:10:39.088775 17990 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:10:39.095151 17990 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e44f622f0b1e4af583ab7e70c159f1f1: Generated new TSK 0
I20260620 00:10:39.095366 17990 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:10:39.097932 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35617
--builtin_ntp_servers=127.0.98.20:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 00:10:39.180190 17999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:10:39.180387 17999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:10:39.180408 17999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:10:39.181613 17999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:10:39.181653 17999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:10:39.181677 17999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:10:39.183135 17999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:10:39.183379 17999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:10:39.183706 17999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:10:39.188730 18007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:39.188760 18005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:39.188738 18004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:39.189054 17999 server_base.cc:1034] running on GCE node
I20260620 00:10:39.189237 17999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:10:39.189527 17999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:10:39.190697 17999 hybrid_clock.cc:648] HybridClock initialized: now 1781914239190682 us; error 30 us; skew 500 ppm
I20260620 00:10:39.191977 17999 webserver.cc:460] Webserver started at http://127.0.98.1:36543/ using document root <none> and password file <none>
I20260620 00:10:39.192251 17999 fs_manager.cc:359] Metadata directory not provided
I20260620 00:10:39.192322 17999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:10:39.192480 17999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:10:39.193602 17999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-0/data/instance:
uuid: "934ff3d8da2a495dacfbfd691caa4631"
format_stamp: "Formatted at 2026-06-20 00:10:39 on dist-test-slave-grqt"
server_key: "7b1195924b900d19508aa9e11400e676"
server_key_iv: "718c891240f117e872b8ce517cef6100"
server_key_version: "encryptionkey@0"
I20260620 00:10:39.193948 17999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-0/wal/instance:
uuid: "934ff3d8da2a495dacfbfd691caa4631"
format_stamp: "Formatted at 2026-06-20 00:10:39 on dist-test-slave-grqt"
server_key: "7b1195924b900d19508aa9e11400e676"
server_key_iv: "718c891240f117e872b8ce517cef6100"
server_key_version: "encryptionkey@0"
I20260620 00:10:39.195284 17999 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:10:39.196110 18013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:39.196316 17999 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:10:39.196411 17999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-0/wal
uuid: "934ff3d8da2a495dacfbfd691caa4631"
format_stamp: "Formatted at 2026-06-20 00:10:39 on dist-test-slave-grqt"
server_key: "7b1195924b900d19508aa9e11400e676"
server_key_iv: "718c891240f117e872b8ce517cef6100"
server_key_version: "encryptionkey@0"
I20260620 00:10:39.196511 17999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:10:39.215260 17999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:10:39.215621 17999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:10:39.215786 17999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:10:39.216035 17999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:10:39.216384 17999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:10:39.216437 17999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:39.216527 17999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:10:39.216553 17999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:39.222245 17999 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38197
I20260620 00:10:39.222290 18126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38197 every 8 connection(s)
I20260620 00:10:39.222622 17999 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:10:39.222775 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 17999
I20260620 00:10:39.222892 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:10:39.223109 392 external_mini_cluster.cc:1428] Setting key 513bbfb861ba27337aa083cb3e2acc5c
I20260620 00:10:39.224256 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35617
--builtin_ntp_servers=127.0.98.20:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 00:10:39.229255 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:39.223734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49051 (local address 127.0.98.62:35617)
0620 00:10:39.223799 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:39.223801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:39.223919 (+ 118us) server_negotiation.cc:407] Connection header received
0620 00:10:39.224698 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:39.224700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:39.224710 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:39.224775 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:39.225750 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:39.226605 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:39.227626 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:39.227871 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:39.227958 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:39.228905 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:39.228907 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:39.228907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:10:39.228913 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:39.228929 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:10:39.228934 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:39.228982 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:39.229123 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 00:10:39.229160 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 00:10:39.229677 18129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:39.223451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35617 (local address 127.0.98.1:49051)
0620 00:10:39.223791 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:39.223835 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:39.224591 (+ 756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:39.224855 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:39.224859 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:39.224870 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:10:39.225630 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:39.225640 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:39.226714 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:39.226715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:39.227537 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:39.227539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:39.227974 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:39.227975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:39.228137 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:39.228749 (+ 612us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:10:39.228750 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:10:39.228779 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:10:39.228804 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:10:39.229027 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:10:39.229032 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:10:39.229037 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:10:39.229124 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 00:10:39.229235 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":97,"threads_started":1}
I20260620 00:10:39.230255 18127 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35617
I20260620 00:10:39.230366 18127 heartbeater.cc:464] Registering TS with master...
I20260620 00:10:39.230559 18127 heartbeater.cc:511] Master 127.0.98.62:35617 requested a full tablet report, sending...
I20260620 00:10:39.231153 17939 ts_manager.cc:194] Registered new tserver with Master: 934ff3d8da2a495dacfbfd691caa4631 (127.0.98.1:38197)
I20260620 00:10:39.232162 17939 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:49051
W20260620 00:10:39.307610 18130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:10:39.307821 18130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:10:39.307843 18130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:10:39.308928 18130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:10:39.308964 18130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:10:39.308987 18130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:10:39.310281 18130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:10:39.310478 18130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:10:39.310722 18130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 00:10:39.315651 18130 server_base.cc:1034] running on GCE node
W20260620 00:10:39.315623 18138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:39.315615 18136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:39.315623 18135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:39.316125 18130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:10:39.316354 18130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:10:39.317507 18130 hybrid_clock.cc:648] HybridClock initialized: now 1781914239317496 us; error 26 us; skew 500 ppm
I20260620 00:10:39.318621 18130 webserver.cc:460] Webserver started at http://127.0.98.2:36663/ using document root <none> and password file <none>
I20260620 00:10:39.318830 18130 fs_manager.cc:359] Metadata directory not provided
I20260620 00:10:39.318871 18130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:10:39.318972 18130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:10:39.320035 18130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-1/data/instance:
uuid: "7c8c9d896eab4759a457a84ceddd23b8"
format_stamp: "Formatted at 2026-06-20 00:10:39 on dist-test-slave-grqt"
server_key: "f4cc8c3aa859053127ec55e64818622e"
server_key_iv: "b2e43b9cc7383bf6d1c60c16e0b5d832"
server_key_version: "encryptionkey@0"
I20260620 00:10:39.320370 18130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7c8c9d896eab4759a457a84ceddd23b8"
format_stamp: "Formatted at 2026-06-20 00:10:39 on dist-test-slave-grqt"
server_key: "f4cc8c3aa859053127ec55e64818622e"
server_key_iv: "b2e43b9cc7383bf6d1c60c16e0b5d832"
server_key_version: "encryptionkey@0"
I20260620 00:10:39.321645 18130 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 00:10:39.322369 18144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:39.322539 18130 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:10:39.322638 18130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-1/wal
uuid: "7c8c9d896eab4759a457a84ceddd23b8"
format_stamp: "Formatted at 2026-06-20 00:10:39 on dist-test-slave-grqt"
server_key: "f4cc8c3aa859053127ec55e64818622e"
server_key_iv: "b2e43b9cc7383bf6d1c60c16e0b5d832"
server_key_version: "encryptionkey@0"
I20260620 00:10:39.322714 18130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:10:39.333043 18130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:10:39.333333 18130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:10:39.333511 18130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:10:39.333734 18130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:10:39.334051 18130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:10:39.334118 18130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:39.334160 18130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:10:39.334187 18130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:39.340131 18130 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36303
I20260620 00:10:39.340179 18257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36303 every 8 connection(s)
I20260620 00:10:39.340533 18130 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:10:39.345620 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:39.341238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36955 (local address 127.0.98.62:35617)
0620 00:10:39.341308 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:39.341311 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:39.341641 (+ 330us) server_negotiation.cc:407] Connection header received
0620 00:10:39.342110 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:39.342112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:39.342124 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:39.342175 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:39.342826 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:39.343320 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:39.344311 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:39.344512 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:39.344566 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:39.345327 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:39.345328 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:39.345328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:10:39.345332 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:39.345345 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:10:39.345348 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:39.345435 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:39.345538 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 00:10:39.345561 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 00:10:39.345834 18260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:39.341238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35617 (local address 127.0.98.2:36955)
0620 00:10:39.341521 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:39.341554 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:39.342025 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:39.342245 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:39.342248 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:39.342256 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:10:39.342744 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:39.342748 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:39.343421 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:39.343421 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:39.344228 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:39.344230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:39.344586 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:39.344587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:39.344690 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:39.345200 (+ 510us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:10:39.345201 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:10:39.345227 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:10:39.345248 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:10:39.345445 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:10:39.345450 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:10:39.345453 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:10:39.345527 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 00:10:39.345604 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":95,"threads_started":1}
I20260620 00:10:39.346294 18258 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35617
I20260620 00:10:39.346369 18258 heartbeater.cc:464] Registering TS with master...
I20260620 00:10:39.346554 18258 heartbeater.cc:511] Master 127.0.98.62:35617 requested a full tablet report, sending...
I20260620 00:10:39.346916 17939 ts_manager.cc:194] Registered new tserver with Master: 7c8c9d896eab4759a457a84ceddd23b8 (127.0.98.2:36303)
I20260620 00:10:39.347262 17939 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:36955
I20260620 00:10:39.350581 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 18130
I20260620 00:10:39.350670 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:10:39.351008 392 external_mini_cluster.cc:1428] Setting key dee6a61082732f1b0dc67fcc62324804
I20260620 00:10:39.352017 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35617
--builtin_ntp_servers=127.0.98.20:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 00:10:39.434028 18261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:10:39.434345 18261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:10:39.434381 18261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:10:39.435662 18261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:10:39.435700 18261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:10:39.435726 18261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:10:39.437002 18261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:10:39.437194 18261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:10:39.437503 18261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:10:39.442950 18269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:39.442978 18266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:39.443015 18267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:39.443060 18261 server_base.cc:1034] running on GCE node
I20260620 00:10:39.443248 18261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:10:39.443499 18261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:10:39.444653 18261 hybrid_clock.cc:648] HybridClock initialized: now 1781914239444637 us; error 41 us; skew 500 ppm
I20260620 00:10:39.446002 18261 webserver.cc:460] Webserver started at http://127.0.98.3:34275/ using document root <none> and password file <none>
I20260620 00:10:39.446257 18261 fs_manager.cc:359] Metadata directory not provided
I20260620 00:10:39.446329 18261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:10:39.446467 18261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:10:39.447527 18261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-2/data/instance:
uuid: "5b1e06ae3c274694ad887985c33148c4"
format_stamp: "Formatted at 2026-06-20 00:10:39 on dist-test-slave-grqt"
server_key: "6687dd7d740173fbd3cebcea231e95f3"
server_key_iv: "188293dc427ac27c19c4a101b94ba58b"
server_key_version: "encryptionkey@0"
I20260620 00:10:39.447881 18261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5b1e06ae3c274694ad887985c33148c4"
format_stamp: "Formatted at 2026-06-20 00:10:39 on dist-test-slave-grqt"
server_key: "6687dd7d740173fbd3cebcea231e95f3"
server_key_iv: "188293dc427ac27c19c4a101b94ba58b"
server_key_version: "encryptionkey@0"
I20260620 00:10:39.449160 18261 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 00:10:39.450016 18275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:39.450305 18261 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:10:39.450404 18261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-2/wal
uuid: "5b1e06ae3c274694ad887985c33148c4"
format_stamp: "Formatted at 2026-06-20 00:10:39 on dist-test-slave-grqt"
server_key: "6687dd7d740173fbd3cebcea231e95f3"
server_key_iv: "188293dc427ac27c19c4a101b94ba58b"
server_key_version: "encryptionkey@0"
I20260620 00:10:39.450485 18261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:10:39.469599 18261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:10:39.469926 18261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:10:39.470088 18261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:10:39.470323 18261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:10:39.470650 18261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:10:39.470707 18261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:39.470749 18261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:10:39.470775 18261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:39.476495 18261 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39535
I20260620 00:10:39.476545 18388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39535 every 8 connection(s)
I20260620 00:10:39.476897 18261 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:10:39.482405 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:39.477634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50663 (local address 127.0.98.62:35617)
0620 00:10:39.477701 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:39.477703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:39.478076 (+ 373us) server_negotiation.cc:407] Connection header received
0620 00:10:39.478617 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:39.478618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:39.478630 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:39.478693 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:39.479445 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:39.479963 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:39.480959 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:39.481169 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:39.481256 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:39.482108 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:39.482109 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:39.482110 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:10:39.482114 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:39.482130 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:10:39.482134 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:39.482197 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:39.482293 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 00:10:39.482324 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 00:10:39.482625 18391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:39.477629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35617 (local address 127.0.98.3:50663)
0620 00:10:39.477897 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:39.477955 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:39.478508 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:39.478768 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:39.478772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:39.478788 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:10:39.479349 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:39.479355 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:39.480064 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:39.480065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:39.480864 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:39.480866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:39.481264 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:39.481265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:39.481418 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:39.481931 (+ 513us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:10:39.481931 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:10:39.481959 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:10:39.481982 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:10:39.482197 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:10:39.482203 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:10:39.482206 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:10:39.482296 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 00:10:39.482398 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":90,"threads_started":1}
I20260620 00:10:39.483165 18389 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35617
I20260620 00:10:39.483244 18389 heartbeater.cc:464] Registering TS with master...
I20260620 00:10:39.483439 18389 heartbeater.cc:511] Master 127.0.98.62:35617 requested a full tablet report, sending...
I20260620 00:10:39.483829 17939 ts_manager.cc:194] Registered new tserver with Master: 5b1e06ae3c274694ad887985c33148c4 (127.0.98.3:39535)
I20260620 00:10:39.484162 17939 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:50663
I20260620 00:10:39.487001 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 18261
I20260620 00:10:39.487083 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:10:39.487236 392 external_mini_cluster.cc:1428] Setting key 4cadf7575e2b59d1f9e496c00934bfd9
I20260620 00:10:39.488081 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:10:39.492458 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:39.489458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47184 (local address 127.0.98.62:35617)
0620 00:10:39.489530 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:39.489531 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:39.489674 (+ 143us) server_negotiation.cc:407] Connection header received
0620 00:10:39.489750 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:39.489751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:39.489763 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:39.489813 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:39.490000 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:39.490476 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:39.491234 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:39.491427 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:39.491516 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:39.492194 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:39.492196 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:39.492196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:10:39.492201 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:39.492221 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:10:39.492225 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:39.492282 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:39.492342 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 00:10:39.492366 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 00:10:39.494390 17939 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:10:39.495860 17939 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:10:39.502552 18401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:39.498546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53684 (local address 127.0.98.2:36303)
0620 00:10:39.498693 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:39.498697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:39.498704 (+ 7us) server_negotiation.cc:407] Connection header received
0620 00:10:39.498735 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:39.498736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:39.498748 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:39.498836 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:10:39.498967 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:39.499918 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:39.501552 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:39.502154 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:39.502241 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:39.502333 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:39.502416 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 00:10:39.502455 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":73,"threads_started":1}
I20260620 00:10:39.502579 18397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:39.498176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36303 (local address 127.0.0.1:53684)
0620 00:10:39.498460 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:39.498469 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:39.498512 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:39.498838 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:39.498841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:39.498843 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:10:39.498910 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:39.498912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:39.500016 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:39.500017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:39.501466 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:39.501469 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:39.502240 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:39.502241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:39.502314 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:39.502326 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 00:10:39.502402 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 00:10:39.502456 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"spinlock_wait_cycles":384,"thread_start_us":101,"threads_started":1}
I20260620 00:10:39.502591 18398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:39.498170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38197 (local address 127.0.0.1:53992)
0620 00:10:39.498345 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:39.498398 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:39.498498 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:39.499290 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:39.499291 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:39.499292 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:10:39.499324 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:39.499325 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:39.500651 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:39.500652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:39.501442 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:39.501445 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:39.502302 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:39.502303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:39.502367 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:39.502387 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 00:10:39.502420 (+ 33us) client_negotiation.cc:240] Negotiation successful
0620 00:10:39.502463 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":70,"threads_started":1}
I20260620 00:10:39.503192 18402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:39.498686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54166 (local address 127.0.98.3:39535)
0620 00:10:39.498840 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:39.498844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:39.498942 (+ 98us) server_negotiation.cc:407] Connection header received
0620 00:10:39.499034 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:39.499036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:39.499052 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:39.499125 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:10:39.499324 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:39.500509 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:39.501795 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:39.502509 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:39.502581 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:39.503010 (+ 429us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:39.503041 (+ 31us) server_negotiation.cc:299] Negotiation successful
0620 00:10:39.503091 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260620 00:10:39.503070 18400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:39.498411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39535 (local address 127.0.0.1:54166)
0620 00:10:39.498862 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:39.498874 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:39.498953 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:39.499194 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:39.499195 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:39.499197 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:10:39.499245 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:39.499247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:39.500599 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:39.500600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:39.501711 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:39.501713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:39.502782 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:39.502782 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:39.502838 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:39.502842 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:10:39.502893 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 00:10:39.502929 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":66,"threads_started":1}
I20260620 00:10:39.503329 18192 tablet_service.cc:1467] Processing CreateTablet for tablet 515caa2cd99443ffa1e752cfbf9e9e96 (DEFAULT_TABLE table=test-table [id=018ed77f153d4a3391a24bfeb2ef8206]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:10:39.503751 18192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 515caa2cd99443ffa1e752cfbf9e9e96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:10:39.504092 18399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:39.498276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53992 (local address 127.0.98.1:38197)
0620 00:10:39.499081 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:39.499085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:39.499095 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:10:39.499130 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:39.499132 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:39.499144 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:39.499412 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:10:39.499473 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:39.500585 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:39.501529 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:39.502199 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:39.503887 (+ 1688us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:39.503974 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:39.503996 (+ 22us) server_negotiation.cc:299] Negotiation successful
0620 00:10:39.504029 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":96,"threads_started":1}
I20260620 00:10:39.504138 18323 tablet_service.cc:1467] Processing CreateTablet for tablet 515caa2cd99443ffa1e752cfbf9e9e96 (DEFAULT_TABLE table=test-table [id=018ed77f153d4a3391a24bfeb2ef8206]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:10:39.504464 18323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 515caa2cd99443ffa1e752cfbf9e9e96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:10:39.506896 18061 tablet_service.cc:1467] Processing CreateTablet for tablet 515caa2cd99443ffa1e752cfbf9e9e96 (DEFAULT_TABLE table=test-table [id=018ed77f153d4a3391a24bfeb2ef8206]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:10:39.507275 18061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 515caa2cd99443ffa1e752cfbf9e9e96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:10:39.507407 18403 tablet_bootstrap.cc:492] T 515caa2cd99443ffa1e752cfbf9e9e96 P 7c8c9d896eab4759a457a84ceddd23b8: Bootstrap starting.
I20260620 00:10:39.507861 18404 tablet_bootstrap.cc:492] T 515caa2cd99443ffa1e752cfbf9e9e96 P 5b1e06ae3c274694ad887985c33148c4: Bootstrap starting.
I20260620 00:10:39.508184 18403 tablet_bootstrap.cc:654] T 515caa2cd99443ffa1e752cfbf9e9e96 P 7c8c9d896eab4759a457a84ceddd23b8: Neither blocks nor log segments found. Creating new log.
I20260620 00:10:39.508490 18404 tablet_bootstrap.cc:654] T 515caa2cd99443ffa1e752cfbf9e9e96 P 5b1e06ae3c274694ad887985c33148c4: Neither blocks nor log segments found. Creating new log.
I20260620 00:10:39.508533 18403 log.cc:824] T 515caa2cd99443ffa1e752cfbf9e9e96 P 7c8c9d896eab4759a457a84ceddd23b8: Log is configured to *not* fsync() on all Append() calls
I20260620 00:10:39.508867 18404 log.cc:824] T 515caa2cd99443ffa1e752cfbf9e9e96 P 5b1e06ae3c274694ad887985c33148c4: Log is configured to *not* fsync() on all Append() calls
I20260620 00:10:39.509336 18403 tablet_bootstrap.cc:492] T 515caa2cd99443ffa1e752cfbf9e9e96 P 7c8c9d896eab4759a457a84ceddd23b8: No bootstrap required, opened a new log
I20260620 00:10:39.509438 18403 ts_tablet_manager.cc:1397] T 515caa2cd99443ffa1e752cfbf9e9e96 P 7c8c9d896eab4759a457a84ceddd23b8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:10:39.509590 18404 tablet_bootstrap.cc:492] T 515caa2cd99443ffa1e752cfbf9e9e96 P 5b1e06ae3c274694ad887985c33148c4: No bootstrap required, opened a new log
I20260620 00:10:39.509668 18404 ts_tablet_manager.cc:1397] T 515caa2cd99443ffa1e752cfbf9e9e96 P 5b1e06ae3c274694ad887985c33148c4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:10:39.510852 18407 tablet_bootstrap.cc:492] T 515caa2cd99443ffa1e752cfbf9e9e96 P 934ff3d8da2a495dacfbfd691caa4631: Bootstrap starting.
I20260620 00:10:39.511536 18407 tablet_bootstrap.cc:654] T 515caa2cd99443ffa1e752cfbf9e9e96 P 934ff3d8da2a495dacfbfd691caa4631: Neither blocks nor log segments found. Creating new log.
I20260620 00:10:39.511854 18407 log.cc:824] T 515caa2cd99443ffa1e752cfbf9e9e96 P 934ff3d8da2a495dacfbfd691caa4631: Log is configured to *not* fsync() on all Append() calls
I20260620 00:10:39.511978 18403 raft_consensus.cc:348] T 515caa2cd99443ffa1e752cfbf9e9e96 P 7c8c9d896eab4759a457a84ceddd23b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934ff3d8da2a495dacfbfd691caa4631" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38197 } } peers { permanent_uuid: "7c8c9d896eab4759a457a84ceddd23b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36303 } } peers { permanent_uuid: "5b1e06ae3c274694ad887985c33148c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39535 } }
I20260620 00:10:39.512051 18404 raft_consensus.cc:348] T 515caa2cd99443ffa1e752cfbf9e9e96 P 5b1e06ae3c274694ad887985c33148c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934ff3d8da2a495dacfbfd691caa4631" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38197 } } peers { permanent_uuid: "7c8c9d896eab4759a457a84ceddd23b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36303 } } peers { permanent_uuid: "5b1e06ae3c274694ad887985c33148c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39535 } }
I20260620 00:10:39.512169 18403 raft_consensus.cc:374] T 515caa2cd99443ffa1e752cfbf9e9e96 P 7c8c9d896eab4759a457a84ceddd23b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:10:39.512202 18404 raft_consensus.cc:374] T 515caa2cd99443ffa1e752cfbf9e9e96 P 5b1e06ae3c274694ad887985c33148c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:10:39.512229 18403 raft_consensus.cc:729] T 515caa2cd99443ffa1e752cfbf9e9e96 P 7c8c9d896eab4759a457a84ceddd23b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c8c9d896eab4759a457a84ceddd23b8, State: Initialized, Role: FOLLOWER
I20260620 00:10:39.512251 18404 raft_consensus.cc:729] T 515caa2cd99443ffa1e752cfbf9e9e96 P 5b1e06ae3c274694ad887985c33148c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b1e06ae3c274694ad887985c33148c4, State: Initialized, Role: FOLLOWER
I20260620 00:10:39.512351 18403 consensus_queue.cc:260] T 515caa2cd99443ffa1e752cfbf9e9e96 P 7c8c9d896eab4759a457a84ceddd23b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934ff3d8da2a495dacfbfd691caa4631" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38197 } } peers { permanent_uuid: "7c8c9d896eab4759a457a84ceddd23b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36303 } } peers { permanent_uuid: "5b1e06ae3c274694ad887985c33148c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39535 } }
I20260620 00:10:39.512372 18404 consensus_queue.cc:260] T 515caa2cd99443ffa1e752cfbf9e9e96 P 5b1e06ae3c274694ad887985c33148c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934ff3d8da2a495dacfbfd691caa4631" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38197 } } peers { permanent_uuid: "7c8c9d896eab4759a457a84ceddd23b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36303 } } peers { permanent_uuid: "5b1e06ae3c274694ad887985c33148c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39535 } }
I20260620 00:10:39.512641 18404 ts_tablet_manager.cc:1428] T 515caa2cd99443ffa1e752cfbf9e9e96 P 5b1e06ae3c274694ad887985c33148c4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:10:39.512633 18403 ts_tablet_manager.cc:1428] T 515caa2cd99443ffa1e752cfbf9e9e96 P 7c8c9d896eab4759a457a84ceddd23b8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:10:39.512702 18258 heartbeater.cc:503] Master 127.0.98.62:35617 was elected leader, sending a full tablet report...
I20260620 00:10:39.512758 18389 heartbeater.cc:503] Master 127.0.98.62:35617 was elected leader, sending a full tablet report...
I20260620 00:10:39.513257 18407 tablet_bootstrap.cc:492] T 515caa2cd99443ffa1e752cfbf9e9e96 P 934ff3d8da2a495dacfbfd691caa4631: No bootstrap required, opened a new log
I20260620 00:10:39.513327 18407 ts_tablet_manager.cc:1397] T 515caa2cd99443ffa1e752cfbf9e9e96 P 934ff3d8da2a495dacfbfd691caa4631: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 00:10:39.515350 18407 raft_consensus.cc:348] T 515caa2cd99443ffa1e752cfbf9e9e96 P 934ff3d8da2a495dacfbfd691caa4631 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934ff3d8da2a495dacfbfd691caa4631" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38197 } } peers { permanent_uuid: "7c8c9d896eab4759a457a84ceddd23b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36303 } } peers { permanent_uuid: "5b1e06ae3c274694ad887985c33148c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39535 } }
I20260620 00:10:39.515539 18407 raft_consensus.cc:374] T 515caa2cd99443ffa1e752cfbf9e9e96 P 934ff3d8da2a495dacfbfd691caa4631 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:10:39.515565 18407 raft_consensus.cc:729] T 515caa2cd99443ffa1e752cfbf9e9e96 P 934ff3d8da2a495dacfbfd691caa4631 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 934ff3d8da2a495dacfbfd691caa4631, State: Initialized, Role: FOLLOWER
I20260620 00:10:39.515698 18407 consensus_queue.cc:260] T 515caa2cd99443ffa1e752cfbf9e9e96 P 934ff3d8da2a495dacfbfd691caa4631 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934ff3d8da2a495dacfbfd691caa4631" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38197 } } peers { permanent_uuid: "7c8c9d896eab4759a457a84ceddd23b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36303 } } peers { permanent_uuid: "5b1e06ae3c274694ad887985c33148c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39535 } }
I20260620 00:10:39.515954 18407 ts_tablet_manager.cc:1428] T 515caa2cd99443ffa1e752cfbf9e9e96 P 934ff3d8da2a495dacfbfd691caa4631: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 00:10:39.516019 18127 heartbeater.cc:503] Master 127.0.98.62:35617 was elected leader, sending a full tablet report...
I20260620 00:10:39.564144 18411 raft_consensus.cc:482] T 515caa2cd99443ffa1e752cfbf9e9e96 P 934ff3d8da2a495dacfbfd691caa4631 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:10:39.564266 18411 raft_consensus.cc:504] T 515caa2cd99443ffa1e752cfbf9e9e96 P 934ff3d8da2a495dacfbfd691caa4631 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934ff3d8da2a495dacfbfd691caa4631" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38197 } } peers { permanent_uuid: "7c8c9d896eab4759a457a84ceddd23b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36303 } } peers { permanent_uuid: "5b1e06ae3c274694ad887985c33148c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39535 } }
I20260620 00:10:39.564723 18411 leader_election.cc:288] T 515caa2cd99443ffa1e752cfbf9e9e96 P 934ff3d8da2a495dacfbfd691caa4631 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c8c9d896eab4759a457a84ceddd23b8 (127.0.98.2:36303), 5b1e06ae3c274694ad887985c33148c4 (127.0.98.3:39535)
I20260620 00:10:39.568709 18401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:39.564950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46233 (local address 127.0.98.2:36303)
0620 00:10:39.564998 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:39.565001 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:39.565013 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:10:39.565074 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:39.565075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:39.565085 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:39.565136 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:10:39.565354 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:39.566167 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:39.567737 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:39.568306 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:39.568392 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:39.568403 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:39.568591 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 00:10:39.568634 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 00:10:39.568709 18402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:39.565027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41635 (local address 127.0.98.3:39535)
0620 00:10:39.565078 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:39.565080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:39.565210 (+ 130us) server_negotiation.cc:407] Connection header received
0620 00:10:39.565252 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:39.565252 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:39.565260 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:39.565297 (+ 37us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:10:39.565460 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:39.566170 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:39.567737 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:39.568324 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:39.568392 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:39.568403 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:39.568591 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 00:10:39.568634 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 00:10:39.568718 18129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:39.564882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36303 (local address 127.0.98.1:46233)
0620 00:10:39.564943 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:39.564956 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:39.565005 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:39.565177 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:39.565177 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:39.565179 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:10:39.565254 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:39.565256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:39.566261 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:39.566262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:39.567594 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:39.567597 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:39.568419 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:39.568420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:39.568504 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:39.568510 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 00:10:39.568572 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 00:10:39.568622 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":16}
I20260620 00:10:39.568725 18412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:39.564915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39535 (local address 127.0.98.1:41635)
0620 00:10:39.565113 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:39.565122 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:39.565199 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:39.565322 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:39.565323 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:39.565324 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:10:39.565411 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:39.565413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:39.566261 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:39.566262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:39.567594 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:39.567597 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:39.568419 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:39.568420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:39.568504 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:39.568511 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 00:10:39.568572 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 00:10:39.568622 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":100,"thread_start_us":135,"threads_started":1}
I20260620 00:10:39.569093 18212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515caa2cd99443ffa1e752cfbf9e9e96" candidate_uuid: "934ff3d8da2a495dacfbfd691caa4631" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c8c9d896eab4759a457a84ceddd23b8" is_pre_election: true
I20260620 00:10:39.569116 18343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515caa2cd99443ffa1e752cfbf9e9e96" candidate_uuid: "934ff3d8da2a495dacfbfd691caa4631" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b1e06ae3c274694ad887985c33148c4" is_pre_election: true
I20260620 00:10:39.569258 18212 raft_consensus.cc:2454] T 515caa2cd99443ffa1e752cfbf9e9e96 P 7c8c9d896eab4759a457a84ceddd23b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 934ff3d8da2a495dacfbfd691caa4631 in term 0.
I20260620 00:10:39.569271 18343 raft_consensus.cc:2454] T 515caa2cd99443ffa1e752cfbf9e9e96 P 5b1e06ae3c274694ad887985c33148c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 934ff3d8da2a495dacfbfd691caa4631 in term 0.
I20260620 00:10:39.569521 18014 leader_election.cc:302] T 515caa2cd99443ffa1e752cfbf9e9e96 P 934ff3d8da2a495dacfbfd691caa4631 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b1e06ae3c274694ad887985c33148c4, 934ff3d8da2a495dacfbfd691caa4631; no voters:
I20260620 00:10:39.569664 18411 raft_consensus.cc:2781] T 515caa2cd99443ffa1e752cfbf9e9e96 P 934ff3d8da2a495dacfbfd691caa4631 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:10:39.569721 18411 raft_consensus.cc:482] T 515caa2cd99443ffa1e752cfbf9e9e96 P 934ff3d8da2a495dacfbfd691caa4631 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:10:39.569739 18411 raft_consensus.cc:3037] T 515caa2cd99443ffa1e752cfbf9e9e96 P 934ff3d8da2a495dacfbfd691caa4631 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:10:39.570499 18411 raft_consensus.cc:504] T 515caa2cd99443ffa1e752cfbf9e9e96 P 934ff3d8da2a495dacfbfd691caa4631 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934ff3d8da2a495dacfbfd691caa4631" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38197 } } peers { permanent_uuid: "7c8c9d896eab4759a457a84ceddd23b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36303 } } peers { permanent_uuid: "5b1e06ae3c274694ad887985c33148c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39535 } }
I20260620 00:10:39.570654 18411 leader_election.cc:288] T 515caa2cd99443ffa1e752cfbf9e9e96 P 934ff3d8da2a495dacfbfd691caa4631 [CANDIDATE]: Term 1 election: Requested vote from peers 7c8c9d896eab4759a457a84ceddd23b8 (127.0.98.2:36303), 5b1e06ae3c274694ad887985c33148c4 (127.0.98.3:39535)
I20260620 00:10:39.570798 18343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515caa2cd99443ffa1e752cfbf9e9e96" candidate_uuid: "934ff3d8da2a495dacfbfd691caa4631" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b1e06ae3c274694ad887985c33148c4"
I20260620 00:10:39.570815 18212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515caa2cd99443ffa1e752cfbf9e9e96" candidate_uuid: "934ff3d8da2a495dacfbfd691caa4631" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c8c9d896eab4759a457a84ceddd23b8"
I20260620 00:10:39.570868 18343 raft_consensus.cc:3037] T 515caa2cd99443ffa1e752cfbf9e9e96 P 5b1e06ae3c274694ad887985c33148c4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:10:39.570875 18212 raft_consensus.cc:3037] T 515caa2cd99443ffa1e752cfbf9e9e96 P 7c8c9d896eab4759a457a84ceddd23b8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:10:39.571554 18212 raft_consensus.cc:2454] T 515caa2cd99443ffa1e752cfbf9e9e96 P 7c8c9d896eab4759a457a84ceddd23b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 934ff3d8da2a495dacfbfd691caa4631 in term 1.
I20260620 00:10:39.571532 18343 raft_consensus.cc:2454] T 515caa2cd99443ffa1e752cfbf9e9e96 P 5b1e06ae3c274694ad887985c33148c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 934ff3d8da2a495dacfbfd691caa4631 in term 1.
I20260620 00:10:39.571720 18014 leader_election.cc:302] T 515caa2cd99443ffa1e752cfbf9e9e96 P 934ff3d8da2a495dacfbfd691caa4631 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b1e06ae3c274694ad887985c33148c4, 934ff3d8da2a495dacfbfd691caa4631; no voters:
I20260620 00:10:39.571808 18411 raft_consensus.cc:2781] T 515caa2cd99443ffa1e752cfbf9e9e96 P 934ff3d8da2a495dacfbfd691caa4631 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:10:39.571955 18411 raft_consensus.cc:686] T 515caa2cd99443ffa1e752cfbf9e9e96 P 934ff3d8da2a495dacfbfd691caa4631 [term 1 LEADER]: Becoming Leader. State: Replica: 934ff3d8da2a495dacfbfd691caa4631, State: Running, Role: LEADER
I20260620 00:10:39.572052 18411 consensus_queue.cc:237] T 515caa2cd99443ffa1e752cfbf9e9e96 P 934ff3d8da2a495dacfbfd691caa4631 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934ff3d8da2a495dacfbfd691caa4631" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38197 } } peers { permanent_uuid: "7c8c9d896eab4759a457a84ceddd23b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36303 } } peers { permanent_uuid: "5b1e06ae3c274694ad887985c33148c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39535 } }
I20260620 00:10:39.572893 17938 catalog_manager.cc:5495] T 515caa2cd99443ffa1e752cfbf9e9e96 P 934ff3d8da2a495dacfbfd691caa4631 reported cstate change: term changed from 0 to 1, leader changed from <none> to 934ff3d8da2a495dacfbfd691caa4631 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "934ff3d8da2a495dacfbfd691caa4631" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934ff3d8da2a495dacfbfd691caa4631" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c8c9d896eab4759a457a84ceddd23b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b1e06ae3c274694ad887985c33148c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39535 } health_report { overall_health: UNKNOWN } } }
W20260620 00:10:39.591126 18259 tablet.cc:2369] T 515caa2cd99443ffa1e752cfbf9e9e96 P 7c8c9d896eab4759a457a84ceddd23b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:10:39.595517 18399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:39.593292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53998 (local address 127.0.98.1:38197)
0620 00:10:39.593348 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:39.593351 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:39.593402 (+ 51us) server_negotiation.cc:407] Connection header received
0620 00:10:39.593451 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:39.593452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:39.593471 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:39.593530 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:10:39.593635 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:39.594116 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:39.594844 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:39.595045 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:39.595100 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:39.595229 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:10:39.595324 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:10:39.595378 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:39.595425 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 00:10:39.595440 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 00:10:39.597452 18212 raft_consensus.cc:1261] T 515caa2cd99443ffa1e752cfbf9e9e96 P 7c8c9d896eab4759a457a84ceddd23b8 [term 1 FOLLOWER]: Refusing update from remote peer 934ff3d8da2a495dacfbfd691caa4631: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:10:39.597455 18343 raft_consensus.cc:1261] T 515caa2cd99443ffa1e752cfbf9e9e96 P 5b1e06ae3c274694ad887985c33148c4 [term 1 FOLLOWER]: Refusing update from remote peer 934ff3d8da2a495dacfbfd691caa4631: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:10:39.597796 18411 consensus_queue.cc:1035] T 515caa2cd99443ffa1e752cfbf9e9e96 P 934ff3d8da2a495dacfbfd691caa4631 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b1e06ae3c274694ad887985c33148c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:10:39.597959 18413 consensus_queue.cc:1035] T 515caa2cd99443ffa1e752cfbf9e9e96 P 934ff3d8da2a495dacfbfd691caa4631 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8c9d896eab4759a457a84ceddd23b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:10:39.600373 18416 mvcc.cc:205] Tried to move back new op lower bound from 7298720725388349440 to 7298720725287784448. Current Snapshot: MvccSnapshot[applied={T|T < 7298720725388349440}]
I20260620 00:10:39.600917 18417 mvcc.cc:205] Tried to move back new op lower bound from 7298720725388349440 to 7298720725287784448. Current Snapshot: MvccSnapshot[applied={T|T < 7298720725388349440}]
I20260620 00:10:39.602892 17937 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:10:39.602995 17937 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:10:39.603780 17937 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e44f622f0b1e4af583ab7e70c159f1f1: Sending DeleteTablet for 3 replicas of tablet 515caa2cd99443ffa1e752cfbf9e9e96
I20260620 00:10:39.604104 18192 tablet_service.cc:1514] Processing DeleteTablet for tablet 515caa2cd99443ffa1e752cfbf9e9e96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:10:39 UTC) from {username='slave'} at 127.0.0.1:53684
I20260620 00:10:39.604121 18061 tablet_service.cc:1514] Processing DeleteTablet for tablet 515caa2cd99443ffa1e752cfbf9e9e96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:10:39 UTC) from {username='slave'} at 127.0.0.1:53992
I20260620 00:10:39.604137 18323 tablet_service.cc:1514] Processing DeleteTablet for tablet 515caa2cd99443ffa1e752cfbf9e9e96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:10:39 UTC) from {username='slave'} at 127.0.0.1:54166
I20260620 00:10:39.604334 18428 tablet_replica.cc:330] T 515caa2cd99443ffa1e752cfbf9e9e96 P 5b1e06ae3c274694ad887985c33148c4: stopping tablet replica
I20260620 00:10:39.604488 18426 tablet_replica.cc:330] T 515caa2cd99443ffa1e752cfbf9e9e96 P 934ff3d8da2a495dacfbfd691caa4631: stopping tablet replica
I20260620 00:10:39.604588 18426 raft_consensus.cc:2229] T 515caa2cd99443ffa1e752cfbf9e9e96 P 934ff3d8da2a495dacfbfd691caa4631 [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:10:39.604646 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 17999
I20260620 00:10:39.604676 18426 raft_consensus.cc:2258] T 515caa2cd99443ffa1e752cfbf9e9e96 P 934ff3d8da2a495dacfbfd691caa4631 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:10:39.605195 18427 tablet_replica.cc:330] T 515caa2cd99443ffa1e752cfbf9e9e96 P 7c8c9d896eab4759a457a84ceddd23b8: stopping tablet replica
I20260620 00:10:39.605309 18427 raft_consensus.cc:2229] T 515caa2cd99443ffa1e752cfbf9e9e96 P 7c8c9d896eab4759a457a84ceddd23b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:10:39.605436 18427 raft_consensus.cc:2258] T 515caa2cd99443ffa1e752cfbf9e9e96 P 7c8c9d896eab4759a457a84ceddd23b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:10:39.606263 18427 ts_tablet_manager.cc:1905] T 515caa2cd99443ffa1e752cfbf9e9e96 P 7c8c9d896eab4759a457a84ceddd23b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:10:39.606588 18428 raft_consensus.cc:2229] T 515caa2cd99443ffa1e752cfbf9e9e96 P 5b1e06ae3c274694ad887985c33148c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:10:39.606710 18428 raft_consensus.cc:2258] T 515caa2cd99443ffa1e752cfbf9e9e96 P 5b1e06ae3c274694ad887985c33148c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:10:39.607095 18428 ts_tablet_manager.cc:1905] T 515caa2cd99443ffa1e752cfbf9e9e96 P 5b1e06ae3c274694ad887985c33148c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:10:39.608210 18427 ts_tablet_manager.cc:1918] T 515caa2cd99443ffa1e752cfbf9e9e96 P 7c8c9d896eab4759a457a84ceddd23b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:10:39.608321 18427 log.cc:1196] T 515caa2cd99443ffa1e752cfbf9e9e96 P 7c8c9d896eab4759a457a84ceddd23b8: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-1/wal/wals/515caa2cd99443ffa1e752cfbf9e9e96
I20260620 00:10:39.608584 18427 ts_tablet_manager.cc:1939] T 515caa2cd99443ffa1e752cfbf9e9e96 P 7c8c9d896eab4759a457a84ceddd23b8: Deleting consensus metadata
I20260620 00:10:39.609030 18428 ts_tablet_manager.cc:1918] T 515caa2cd99443ffa1e752cfbf9e9e96 P 5b1e06ae3c274694ad887985c33148c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:10:39.609105 18428 log.cc:1196] T 515caa2cd99443ffa1e752cfbf9e9e96 P 5b1e06ae3c274694ad887985c33148c4: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914001714546-392-0/minicluster-data/ts-2/wal/wals/515caa2cd99443ffa1e752cfbf9e9e96
I20260620 00:10:39.609071 17924 catalog_manager.cc:4841] TS 7c8c9d896eab4759a457a84ceddd23b8 (127.0.98.2:36303): tablet 515caa2cd99443ffa1e752cfbf9e9e96 (table test-table [id=018ed77f153d4a3391a24bfeb2ef8206]) successfully deleted
I20260620 00:10:39.609347 18428 ts_tablet_manager.cc:1939] T 515caa2cd99443ffa1e752cfbf9e9e96 P 5b1e06ae3c274694ad887985c33148c4: Deleting consensus metadata
I20260620 00:10:39.609951 17923 catalog_manager.cc:4841] TS 5b1e06ae3c274694ad887985c33148c4 (127.0.98.3:39535): tablet 515caa2cd99443ffa1e752cfbf9e9e96 (table test-table [id=018ed77f153d4a3391a24bfeb2ef8206]) successfully deleted
W20260620 00:10:39.610983 17923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38197 (error 108)
I20260620 00:10:39.611199 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 18130
I20260620 00:10:39.611784 18398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:39.611377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38197 (local address 127.0.0.1:54002)
0620 00:10:39.611704 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:39.611744 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306}
W20260620 00:10:39.611931 17923 catalog_manager.cc:4584] TS 934ff3d8da2a495dacfbfd691caa4631 (127.0.98.1:38197): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 515caa2cd99443ffa1e752cfbf9e9e96: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38197: connect: Connection refused (error 111)
I20260620 00:10:39.616577 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 18261
I20260620 00:10:39.621168 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 17908
2026-06-20T00:10:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (686 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:10:39 dist-test-slave-grqt krb5kdc[18432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:10:39 dist-test-slave-grqt krb5kdc[18432](info): set up 2 sockets
Jun 20 00:10:39 dist-test-slave-grqt krb5kdc[18432](info): commencing operation
krb5kdc: starting...
W20260620 00:10:40.665047 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:10:40 dist-test-slave-grqt krb5kdc[18432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:10:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:10:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:10:40.784601 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43457
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43457
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:10:40.867501 18445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:10:40.867765 18445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:10:40.867802 18445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:10:40.869006 18445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:10:40.869082 18445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:10:40.869107 18445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:10:40.869128 18445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:10:40.869158 18445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:10:40.870766 18445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:10:40.871069 18445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:10:40.871306 18445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:10:40.876332 18450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:40.876338 18453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:40.876402 18451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:40.876678 18445 server_base.cc:1034] running on GCE node
I20260620 00:10:40.876959 18445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:10:40.877254 18445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:10:40.878412 18445 hybrid_clock.cc:648] HybridClock initialized: now 1781914240878407 us; error 46 us; skew 500 ppm
Jun 20 00:10:40 dist-test-slave-grqt krb5kdc[18432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:10:40.881007 18445 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:10:40.881538 18445 webserver.cc:460] Webserver started at http://127.0.98.62:33845/ using document root <none> and password file <none>
I20260620 00:10:40.881785 18445 fs_manager.cc:359] Metadata directory not provided
I20260620 00:10:40.881853 18445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:10:40.882021 18445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:10:40.883129 18445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "37858f2a7eda4485b62c4fbd4e9736a0"
format_stamp: "Formatted at 2026-06-20 00:10:40 on dist-test-slave-grqt"
server_key: "4ef7cdf513e644a047160026ec6d1c14"
server_key_iv: "b3dd678fab9d91812fc3e74b43844e1b"
server_key_version: "encryptionkey@0"
I20260620 00:10:40.883517 18445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "37858f2a7eda4485b62c4fbd4e9736a0"
format_stamp: "Formatted at 2026-06-20 00:10:40 on dist-test-slave-grqt"
server_key: "4ef7cdf513e644a047160026ec6d1c14"
server_key_iv: "b3dd678fab9d91812fc3e74b43844e1b"
server_key_version: "encryptionkey@0"
I20260620 00:10:40.884954 18445 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:10:40.885875 18460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:40.886099 18445 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:10:40.886206 18445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "37858f2a7eda4485b62c4fbd4e9736a0"
format_stamp: "Formatted at 2026-06-20 00:10:40 on dist-test-slave-grqt"
server_key: "4ef7cdf513e644a047160026ec6d1c14"
server_key_iv: "b3dd678fab9d91812fc3e74b43844e1b"
server_key_version: "encryptionkey@0"
I20260620 00:10:40.886297 18445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:10:40.897429 18445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:10:40.899577 18445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:10:40.899762 18445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:10:40.904204 18445 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43457
I20260620 00:10:40.904242 18512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43457 every 8 connection(s)
I20260620 00:10:40.904613 18445 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:10:40.905584 18513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:10:40.907896 18513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37858f2a7eda4485b62c4fbd4e9736a0: Bootstrap starting.
I20260620 00:10:40.908439 18513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37858f2a7eda4485b62c4fbd4e9736a0: Neither blocks nor log segments found. Creating new log.
I20260620 00:10:40.908689 18513 log.cc:824] T 00000000000000000000000000000000 P 37858f2a7eda4485b62c4fbd4e9736a0: Log is configured to *not* fsync() on all Append() calls
I20260620 00:10:40.909569 18513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37858f2a7eda4485b62c4fbd4e9736a0: No bootstrap required, opened a new log
I20260620 00:10:40.909842 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 18445
I20260620 00:10:40.909976 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:10:40.910180 392 external_mini_cluster.cc:1428] Setting key 64dde7df39cc6e8a6d3c2a0cc647363e
I20260620 00:10:40.911751 18513 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37858f2a7eda4485b62c4fbd4e9736a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37858f2a7eda4485b62c4fbd4e9736a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43457 } }
I20260620 00:10:40.911927 18513 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37858f2a7eda4485b62c4fbd4e9736a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:10:40.911968 18513 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37858f2a7eda4485b62c4fbd4e9736a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37858f2a7eda4485b62c4fbd4e9736a0, State: Initialized, Role: FOLLOWER
I20260620 00:10:40.912155 18513 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37858f2a7eda4485b62c4fbd4e9736a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37858f2a7eda4485b62c4fbd4e9736a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43457 } }
I20260620 00:10:40.912256 18513 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37858f2a7eda4485b62c4fbd4e9736a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:10:40.912292 18513 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37858f2a7eda4485b62c4fbd4e9736a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:10:40.912348 18513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37858f2a7eda4485b62c4fbd4e9736a0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:10:40.913480 18513 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37858f2a7eda4485b62c4fbd4e9736a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37858f2a7eda4485b62c4fbd4e9736a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43457 } }
I20260620 00:10:40.913589 18513 leader_election.cc:302] T 00000000000000000000000000000000 P 37858f2a7eda4485b62c4fbd4e9736a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37858f2a7eda4485b62c4fbd4e9736a0; no voters:
I20260620 00:10:40.913775 18513 leader_election.cc:288] T 00000000000000000000000000000000 P 37858f2a7eda4485b62c4fbd4e9736a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:10:40.913805 18518 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37858f2a7eda4485b62c4fbd4e9736a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:10:40.913904 18518 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37858f2a7eda4485b62c4fbd4e9736a0 [term 1 LEADER]: Becoming Leader. State: Replica: 37858f2a7eda4485b62c4fbd4e9736a0, State: Running, Role: LEADER
I20260620 00:10:40.914052 18513 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37858f2a7eda4485b62c4fbd4e9736a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:10:40.914058 18518 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37858f2a7eda4485b62c4fbd4e9736a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37858f2a7eda4485b62c4fbd4e9736a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43457 } }
I20260620 00:10:40.914510 18520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37858f2a7eda4485b62c4fbd4e9736a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37858f2a7eda4485b62c4fbd4e9736a0. Latest consensus state: current_term: 1 leader_uuid: "37858f2a7eda4485b62c4fbd4e9736a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37858f2a7eda4485b62c4fbd4e9736a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43457 } } }
I20260620 00:10:40.914661 18520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37858f2a7eda4485b62c4fbd4e9736a0 [sys.catalog]: This master's current role is: LEADER
I20260620 00:10:40.914849 18519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37858f2a7eda4485b62c4fbd4e9736a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37858f2a7eda4485b62c4fbd4e9736a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37858f2a7eda4485b62c4fbd4e9736a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43457 } } }
I20260620 00:10:40.914976 18519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37858f2a7eda4485b62c4fbd4e9736a0 [sys.catalog]: This master's current role is: LEADER
I20260620 00:10:40.915153 18526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:10:40.915694 18526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:10:40.917477 18526 catalog_manager.cc:1347] Generated new cluster ID: 7d8c5eb6cf7f482d8cbcf08adba62c81
I20260620 00:10:40.917557 18526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 00:10:40 dist-test-slave-grqt krb5kdc[18432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:10:40.923257 18517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:40.910767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41014 (local address 127.0.98.62:43457)
0620 00:10:40.910949 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:40.910959 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:40.910996 (+ 37us) server_negotiation.cc:407] Connection header received
0620 00:10:40.911732 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:40.911739 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:40.911794 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:40.911941 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:40.912758 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:40.913846 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:40.914953 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:40.915564 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:40.915676 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:40.918640 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:40.918644 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:40.918645 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:10:40.918656 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:40.920834 (+ 2178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:40.921273 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:40.921276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:40.921277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:40.921433 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:40.921764 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:40.921765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:40.921765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:40.922145 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:40.922765 (+ 620us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:40.922842 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 00:10:40.923002 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
I20260620 00:10:40.932155 18526 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:10:40.932960 18526 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:10:40.944128 18526 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37858f2a7eda4485b62c4fbd4e9736a0: Generated new TSK 0
I20260620 00:10:40.944346 18526 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:10:40.972922 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43457
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:10:41.054872 18539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:10:41.055088 18539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:10:41.055114 18539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:10:41.056258 18539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:10:41.056299 18539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:10:41.056325 18539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:10:41.057806 18539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:10:41.058065 18539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:10:41.058300 18539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:10:41.063306 18544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:41.063329 18545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:41.063313 18547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:41.063428 18539 server_base.cc:1034] running on GCE node
I20260620 00:10:41.063678 18539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:10:41.063917 18539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:10:41.065093 18539 hybrid_clock.cc:648] HybridClock initialized: now 1781914241065082 us; error 30 us; skew 500 ppm
Jun 20 00:10:41 dist-test-slave-grqt krb5kdc[18432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:10:41.067629 18539 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:10:41.068143 18539 webserver.cc:460] Webserver started at http://127.0.98.1:38315/ using document root <none> and password file <none>
I20260620 00:10:41.068374 18539 fs_manager.cc:359] Metadata directory not provided
I20260620 00:10:41.068440 18539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:10:41.068578 18539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:10:41.069696 18539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-0/data/instance:
uuid: "2d958665cb98413cb4da11c123d4b1ad"
format_stamp: "Formatted at 2026-06-20 00:10:41 on dist-test-slave-grqt"
server_key: "99399b3090b3d8211bb80732b04faa98"
server_key_iv: "39bf3b50d3b27b52491cd50694b49856"
server_key_version: "encryptionkey@0"
I20260620 00:10:41.070076 18539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2d958665cb98413cb4da11c123d4b1ad"
format_stamp: "Formatted at 2026-06-20 00:10:41 on dist-test-slave-grqt"
server_key: "99399b3090b3d8211bb80732b04faa98"
server_key_iv: "39bf3b50d3b27b52491cd50694b49856"
server_key_version: "encryptionkey@0"
I20260620 00:10:41.071384 18539 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:10:41.072121 18554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:41.072301 18539 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:10:41.072407 18539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-0/wal
uuid: "2d958665cb98413cb4da11c123d4b1ad"
format_stamp: "Formatted at 2026-06-20 00:10:41 on dist-test-slave-grqt"
server_key: "99399b3090b3d8211bb80732b04faa98"
server_key_iv: "39bf3b50d3b27b52491cd50694b49856"
server_key_version: "encryptionkey@0"
I20260620 00:10:41.072484 18539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:10:41.096410 18539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:10:41.098875 18539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:10:41.099005 18539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:10:41.099200 18539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:10:41.099453 18539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:10:41.099483 18539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:41.099506 18539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:10:41.099519 18539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:41.105696 18539 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45001
I20260620 00:10:41.105835 18667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45001 every 8 connection(s)
I20260620 00:10:41.106130 18539 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:10:41.108409 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 18539
I20260620 00:10:41.108544 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:10:41.108760 392 external_mini_cluster.cc:1428] Setting key b313b11aba99f20b31922d189a6580b2
Jun 20 00:10:41 dist-test-slave-grqt krb5kdc[18432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 00:10:41.117883 18517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:41.106899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59907 (local address 127.0.98.62:43457)
0620 00:10:41.106970 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:41.106972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:41.107213 (+ 241us) server_negotiation.cc:407] Connection header received
0620 00:10:41.108188 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:41.108189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:41.108223 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:41.108288 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:41.109293 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:41.109952 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:41.111525 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:41.111874 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:41.111954 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:41.114705 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:41.114707 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:41.114707 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:10:41.114716 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:41.116364 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:41.116812 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:41.116814 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:41.116814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:41.116879 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:41.117154 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:41.117155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:41.117156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:41.117277 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:41.117355 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:41.117674 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 00:10:41.117789 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 00:10:41.118059 18670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:41.106884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43457 (local address 127.0.98.1:59907)
0620 00:10:41.107055 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:41.107098 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:41.108064 (+ 966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:41.108443 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:41.108447 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:41.108856 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:10:41.109178 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:41.109185 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:41.110055 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:41.110056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:41.111393 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:41.111396 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:41.111971 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:41.111972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:41.112167 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:41.112779 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:10:41.112791 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:10:41.114546 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:10:41.116472 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:10:41.116479 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:10:41.116481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:10:41.116717 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:10:41.116961 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:10:41.116962 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:10:41.116963 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:10:41.117085 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:10:41.117347 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:10:41.117353 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:10:41.117538 (+ 185us) client_negotiation.cc:769] Sending connection context
0620 00:10:41.117653 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 00:10:41.117816 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":99,"threads_started":1}
I20260620 00:10:41.118693 18668 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43457
I20260620 00:10:41.118791 18668 heartbeater.cc:464] Registering TS with master...
I20260620 00:10:41.118961 18668 heartbeater.cc:511] Master 127.0.98.62:43457 requested a full tablet report, sending...
I20260620 00:10:41.119580 18477 ts_manager.cc:194] Registered new tserver with Master: 2d958665cb98413cb4da11c123d4b1ad (127.0.98.1:45001)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 00:10:41.120298 18477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59907
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:10:41.138449 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43457
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:10:41.221796 18673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:10:41.222036 18673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:10:41.222062 18673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:10:41.223300 18673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:10:41.223351 18673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:10:41.223380 18673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:10:41.224980 18673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:10:41.225251 18673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:10:41.225524 18673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:10:41.230661 18679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:41.230808 18673 server_base.cc:1034] running on GCE node
W20260620 00:10:41.230654 18681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:41.230669 18678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:41.231129 18673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:10:41.231389 18673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:10:41.232565 18673 hybrid_clock.cc:648] HybridClock initialized: now 1781914241232555 us; error 42 us; skew 500 ppm
Jun 20 00:10:41 dist-test-slave-grqt krb5kdc[18432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:10:41.235224 18673 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:10:41.235630 18673 webserver.cc:460] Webserver started at http://127.0.98.2:46767/ using document root <none> and password file <none>
I20260620 00:10:41.235817 18673 fs_manager.cc:359] Metadata directory not provided
I20260620 00:10:41.235858 18673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:10:41.235946 18673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:10:41.236974 18673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-1/data/instance:
uuid: "5586c77db5354d00ad3eef8f8f4476e8"
format_stamp: "Formatted at 2026-06-20 00:10:41 on dist-test-slave-grqt"
server_key: "30d84923830d7f9099807492c8007a43"
server_key_iv: "ca7bb02bdc80a9f7a455cf095b8d7b2d"
server_key_version: "encryptionkey@0"
I20260620 00:10:41.237262 18673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5586c77db5354d00ad3eef8f8f4476e8"
format_stamp: "Formatted at 2026-06-20 00:10:41 on dist-test-slave-grqt"
server_key: "30d84923830d7f9099807492c8007a43"
server_key_iv: "ca7bb02bdc80a9f7a455cf095b8d7b2d"
server_key_version: "encryptionkey@0"
I20260620 00:10:41.238669 18673 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:10:41.239357 18688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:41.239562 18673 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:10:41.239651 18673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-1/wal
uuid: "5586c77db5354d00ad3eef8f8f4476e8"
format_stamp: "Formatted at 2026-06-20 00:10:41 on dist-test-slave-grqt"
server_key: "30d84923830d7f9099807492c8007a43"
server_key_iv: "ca7bb02bdc80a9f7a455cf095b8d7b2d"
server_key_version: "encryptionkey@0"
I20260620 00:10:41.239709 18673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:10:41.257103 18673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:10:41.259505 18673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:10:41.259727 18673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:10:41.260025 18673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:10:41.260335 18673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:10:41.260380 18673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:41.260418 18673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:10:41.260438 18673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:41.266707 18673 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41297
I20260620 00:10:41.266788 18801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41297 every 8 connection(s)
I20260620 00:10:41.267150 18673 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 00:10:41 dist-test-slave-grqt krb5kdc[18432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:10:41.274124 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 18673
I20260620 00:10:41.274240 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:10:41.274452 392 external_mini_cluster.cc:1428] Setting key 1af26309a92755bab3aa5eb8e22a5069
I20260620 00:10:41.277519 18517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:41.267974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47715 (local address 127.0.98.62:43457)
0620 00:10:41.268034 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:41.268036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:41.268206 (+ 170us) server_negotiation.cc:407] Connection header received
0620 00:10:41.269224 (+ 1018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:41.269225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:41.269236 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:41.269299 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:41.269994 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:41.270492 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:41.271468 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:41.271659 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:41.271718 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:41.273918 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:41.273919 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:41.273920 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:10:41.273926 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:41.275833 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:41.276377 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:41.276378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:41.276379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:41.276445 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:41.276710 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:41.276711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:41.276712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:41.276869 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:41.276946 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:41.277323 (+ 377us) server_negotiation.cc:299] Negotiation successful
0620 00:10:41.277448 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 00:10:41.277850 18804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:41.267879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43457 (local address 127.0.98.2:47715)
0620 00:10:41.268055 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:41.268135 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:41.269106 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:41.269422 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:41.269427 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:41.269704 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:10:41.269899 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:41.269904 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:41.270573 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:41.270574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:41.271387 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:41.271389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:41.271738 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:41.271739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:41.271851 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:41.272403 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:10:41.272412 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:10:41.273828 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:10:41.275923 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:10:41.275929 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:10:41.275931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:10:41.276213 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:10:41.276519 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:10:41.276520 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:10:41.276520 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:10:41.276640 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:10:41.276956 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:10:41.276963 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:10:41.277161 (+ 198us) client_negotiation.cc:769] Sending connection context
0620 00:10:41.277302 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 00:10:41.277546 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":101,"threads_started":1}
I20260620 00:10:41.278441 18802 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43457
I20260620 00:10:41.278558 18802 heartbeater.cc:464] Registering TS with master...
I20260620 00:10:41.278764 18802 heartbeater.cc:511] Master 127.0.98.62:43457 requested a full tablet report, sending...
I20260620 00:10:41.279167 18477 ts_manager.cc:194] Registered new tserver with Master: 5586c77db5354d00ad3eef8f8f4476e8 (127.0.98.2:41297)
I20260620 00:10:41.279556 18477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47715
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:10:41.303640 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43457
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:10:41.386660 18807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:10:41.386865 18807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:10:41.386889 18807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:10:41.388005 18807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:10:41.388044 18807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:10:41.388072 18807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:10:41.389650 18807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:10:41.389873 18807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:10:41.390054 18807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:10:41.395138 18815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:41.395150 18812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:41.395198 18807 server_base.cc:1034] running on GCE node
W20260620 00:10:41.395151 18813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:41.395539 18807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:10:41.395821 18807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:10:41.397002 18807 hybrid_clock.cc:648] HybridClock initialized: now 1781914241396981 us; error 38 us; skew 500 ppm
Jun 20 00:10:41 dist-test-slave-grqt krb5kdc[18432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:10:41.399675 18807 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:10:41.400104 18807 webserver.cc:460] Webserver started at http://127.0.98.3:37049/ using document root <none> and password file <none>
I20260620 00:10:41.400337 18807 fs_manager.cc:359] Metadata directory not provided
I20260620 00:10:41.400384 18807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:10:41.400516 18807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:10:41.401736 18807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-2/data/instance:
uuid: "d113821a62ff45ee8cf6e4ab0ceeb0b2"
format_stamp: "Formatted at 2026-06-20 00:10:41 on dist-test-slave-grqt"
server_key: "657bc773ffe5692765ed77ad65cbca20"
server_key_iv: "985ba8c9a4a02186f2b2b5f93433aa89"
server_key_version: "encryptionkey@0"
I20260620 00:10:41.402129 18807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d113821a62ff45ee8cf6e4ab0ceeb0b2"
format_stamp: "Formatted at 2026-06-20 00:10:41 on dist-test-slave-grqt"
server_key: "657bc773ffe5692765ed77ad65cbca20"
server_key_iv: "985ba8c9a4a02186f2b2b5f93433aa89"
server_key_version: "encryptionkey@0"
I20260620 00:10:41.403465 18807 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 00:10:41.404196 18822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:41.404400 18807 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:10:41.404495 18807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-2/wal
uuid: "d113821a62ff45ee8cf6e4ab0ceeb0b2"
format_stamp: "Formatted at 2026-06-20 00:10:41 on dist-test-slave-grqt"
server_key: "657bc773ffe5692765ed77ad65cbca20"
server_key_iv: "985ba8c9a4a02186f2b2b5f93433aa89"
server_key_version: "encryptionkey@0"
I20260620 00:10:41.404574 18807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:10:41.414891 18807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:10:41.417296 18807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:10:41.417479 18807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:10:41.417670 18807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:10:41.417922 18807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:10:41.417951 18807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:41.417973 18807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:10:41.417986 18807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:41.423600 18807 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34251
I20260620 00:10:41.423673 18935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34251 every 8 connection(s)
I20260620 00:10:41.424021 18807 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:10:41.428577 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 18807
I20260620 00:10:41.428664 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:10:41.428866 392 external_mini_cluster.cc:1428] Setting key 4f51ed59d5cf430d4fc75d874fe1e00a
Jun 20 00:10:41 dist-test-slave-grqt krb5kdc[18432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:10:41.433575 18517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:41.424819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53515 (local address 127.0.98.62:43457)
0620 00:10:41.424929 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:41.424931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:41.425095 (+ 164us) server_negotiation.cc:407] Connection header received
0620 00:10:41.426162 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:41.426163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:41.426175 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:41.426234 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:41.426877 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:41.427390 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:41.428428 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:41.428623 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:41.428710 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:41.431012 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:41.431013 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:41.431013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:10:41.431020 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:41.432326 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:41.432702 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:41.432703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:41.432703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:41.432751 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:41.432997 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:41.432998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:41.432998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:41.433102 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:41.433153 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:41.433443 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 00:10:41.433513 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 00:10:41.433786 18938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:41.424749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43457 (local address 127.0.98.3:53515)
0620 00:10:41.424955 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:41.425007 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:41.426048 (+ 1041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:41.426313 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:41.426317 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:41.426591 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:10:41.426782 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:41.426786 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:41.427487 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:41.427488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:41.428301 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:41.428303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:41.428743 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:41.428744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:41.428903 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:41.429541 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:10:41.429550 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:10:41.430928 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:10:41.432406 (+ 1478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:10:41.432413 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:10:41.432416 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:10:41.432634 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:10:41.432826 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:10:41.432827 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:10:41.432827 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:10:41.432933 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:10:41.433167 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:10:41.433171 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:10:41.433287 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 00:10:41.433427 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 00:10:41.433566 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":80,"threads_started":1}
I20260620 00:10:41.434324 18936 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43457
I20260620 00:10:41.434435 18936 heartbeater.cc:464] Registering TS with master...
I20260620 00:10:41.434660 18936 heartbeater.cc:511] Master 127.0.98.62:43457 requested a full tablet report, sending...
I20260620 00:10:41.435112 18477 ts_manager.cc:194] Registered new tserver with Master: d113821a62ff45ee8cf6e4ab0ceeb0b2 (127.0.98.3:34251)
I20260620 00:10:41.435513 18477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53515
I20260620 00:10:41.440614 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:10:41.447911 18517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:41.441964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41020 (local address 127.0.98.62:43457)
0620 00:10:41.442030 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:41.442032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:41.442109 (+ 77us) server_negotiation.cc:407] Connection header received
0620 00:10:41.442177 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:41.442177 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:41.442185 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:41.442225 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:41.442828 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:41.443303 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:41.444109 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:41.444373 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:41.444433 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:41.445799 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:41.445800 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:41.445800 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:10:41.445806 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:41.446908 (+ 1102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:41.447267 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:41.447268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:41.447268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:41.447314 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:41.447501 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:41.447501 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:41.447501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:41.447617 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:41.447696 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:41.447801 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 00:10:41.447848 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 00:10:41.449478 18477 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:10:41.451048 18477 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:10:41.457885 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:41.453708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41297 (local address 127.0.0.1:39336)
0620 00:10:41.454350 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:41.454361 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:41.454413 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:41.454664 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:41.454665 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:41.454667 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:10:41.454699 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:41.454701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:41.455863 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:41.455864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:41.456793 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:41.456795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:41.457570 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:41.457571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:41.457652 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:41.457666 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 00:10:41.457737 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 00:10:41.457793 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"mutex_wait_us":21,"thread_start_us":69,"threads_started":1}
I20260620 00:10:41.458119 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:41.453674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40382 (local address 127.0.98.1:45001)
0620 00:10:41.454092 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:41.454095 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:41.454103 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:10:41.454141 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:41.454146 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:41.454158 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:41.454243 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:10:41.454407 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:41.455507 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:41.456890 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:41.457649 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:41.457710 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:41.457763 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:41.458011 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 00:10:41.458051 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":96,"threads_started":1}
I20260620 00:10:41.457973 18952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:41.453847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37686 (local address 127.0.98.3:34251)
0620 00:10:41.454041 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:41.454046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:41.454059 (+ 13us) server_negotiation.cc:407] Connection header received
0620 00:10:41.454101 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:41.454103 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:41.454117 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:41.454222 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:10:41.454596 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:41.455326 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:41.456889 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:41.457630 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:41.457713 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:41.457763 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:41.457845 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 00:10:41.457887 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":78,"threads_started":1}
I20260620 00:10:41.458216 18951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:41.453835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39336 (local address 127.0.98.2:41297)
0620 00:10:41.454051 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:41.454055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:41.454415 (+ 360us) server_negotiation.cc:407] Connection header received
0620 00:10:41.454487 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:41.454490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:41.454506 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:41.454577 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:10:41.454782 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:41.455759 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:41.456878 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:41.457475 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:41.457567 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:41.457629 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:41.458077 (+ 448us) server_negotiation.cc:299] Negotiation successful
0620 00:10:41.458121 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":81,"threads_started":1}
I20260620 00:10:41.457969 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:41.453498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34251 (local address 127.0.0.1:37686)
0620 00:10:41.453657 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:41.453698 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:41.453762 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:41.454228 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:41.454232 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:41.454235 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:10:41.454309 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:41.454311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:41.455429 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:41.455430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:41.456785 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:41.456788 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:41.457716 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:41.457717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:41.457774 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:41.457776 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 00:10:41.457821 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 00:10:41.457854 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":37,"thread_start_us":50,"threads_started":1}
I20260620 00:10:41.458071 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:41.453505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45001 (local address 127.0.0.1:40382)
0620 00:10:41.453858 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:41.453870 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:41.453930 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:41.454257 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:41.454259 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:41.454261 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:10:41.454320 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:41.454322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:41.455609 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:41.455610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:41.456799 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:41.456801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:41.457801 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:41.457802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:41.457866 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:41.457870 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:10:41.457918 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 00:10:41.457958 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":11,"thread_start_us":98,"threads_started":1}
I20260620 00:10:41.459182 18736 tablet_service.cc:1467] Processing CreateTablet for tablet 36e9521e864d43a79d3450570cb82c59 (DEFAULT_TABLE table=test-table [id=b24c9375b1f24cf1bce023f812ecae72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:10:41.459333 18870 tablet_service.cc:1467] Processing CreateTablet for tablet 36e9521e864d43a79d3450570cb82c59 (DEFAULT_TABLE table=test-table [id=b24c9375b1f24cf1bce023f812ecae72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:10:41.459575 18736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36e9521e864d43a79d3450570cb82c59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:10:41.459605 18870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36e9521e864d43a79d3450570cb82c59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:10:41.461867 18602 tablet_service.cc:1467] Processing CreateTablet for tablet 36e9521e864d43a79d3450570cb82c59 (DEFAULT_TABLE table=test-table [id=b24c9375b1f24cf1bce023f812ecae72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:10:41.462245 18602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36e9521e864d43a79d3450570cb82c59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:10:41.463079 18954 tablet_bootstrap.cc:492] T 36e9521e864d43a79d3450570cb82c59 P d113821a62ff45ee8cf6e4ab0ceeb0b2: Bootstrap starting.
I20260620 00:10:41.463686 18954 tablet_bootstrap.cc:654] T 36e9521e864d43a79d3450570cb82c59 P d113821a62ff45ee8cf6e4ab0ceeb0b2: Neither blocks nor log segments found. Creating new log.
I20260620 00:10:41.464188 18954 log.cc:824] T 36e9521e864d43a79d3450570cb82c59 P d113821a62ff45ee8cf6e4ab0ceeb0b2: Log is configured to *not* fsync() on all Append() calls
I20260620 00:10:41.464851 18954 tablet_bootstrap.cc:492] T 36e9521e864d43a79d3450570cb82c59 P d113821a62ff45ee8cf6e4ab0ceeb0b2: No bootstrap required, opened a new log
I20260620 00:10:41.464906 18954 ts_tablet_manager.cc:1397] T 36e9521e864d43a79d3450570cb82c59 P d113821a62ff45ee8cf6e4ab0ceeb0b2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:10:41.465273 18956 tablet_bootstrap.cc:492] T 36e9521e864d43a79d3450570cb82c59 P 2d958665cb98413cb4da11c123d4b1ad: Bootstrap starting.
I20260620 00:10:41.465417 18953 tablet_bootstrap.cc:492] T 36e9521e864d43a79d3450570cb82c59 P 5586c77db5354d00ad3eef8f8f4476e8: Bootstrap starting.
I20260620 00:10:41.466068 18956 tablet_bootstrap.cc:654] T 36e9521e864d43a79d3450570cb82c59 P 2d958665cb98413cb4da11c123d4b1ad: Neither blocks nor log segments found. Creating new log.
I20260620 00:10:41.466068 18953 tablet_bootstrap.cc:654] T 36e9521e864d43a79d3450570cb82c59 P 5586c77db5354d00ad3eef8f8f4476e8: Neither blocks nor log segments found. Creating new log.
I20260620 00:10:41.466677 18954 raft_consensus.cc:348] T 36e9521e864d43a79d3450570cb82c59 P d113821a62ff45ee8cf6e4ab0ceeb0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d113821a62ff45ee8cf6e4ab0ceeb0b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34251 } } peers { permanent_uuid: "2d958665cb98413cb4da11c123d4b1ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45001 } } peers { permanent_uuid: "5586c77db5354d00ad3eef8f8f4476e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41297 } }
I20260620 00:10:41.466781 18953 log.cc:824] T 36e9521e864d43a79d3450570cb82c59 P 5586c77db5354d00ad3eef8f8f4476e8: Log is configured to *not* fsync() on all Append() calls
I20260620 00:10:41.466781 18956 log.cc:824] T 36e9521e864d43a79d3450570cb82c59 P 2d958665cb98413cb4da11c123d4b1ad: Log is configured to *not* fsync() on all Append() calls
I20260620 00:10:41.466836 18954 raft_consensus.cc:374] T 36e9521e864d43a79d3450570cb82c59 P d113821a62ff45ee8cf6e4ab0ceeb0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:10:41.466881 18954 raft_consensus.cc:729] T 36e9521e864d43a79d3450570cb82c59 P d113821a62ff45ee8cf6e4ab0ceeb0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d113821a62ff45ee8cf6e4ab0ceeb0b2, State: Initialized, Role: FOLLOWER
I20260620 00:10:41.467005 18954 consensus_queue.cc:260] T 36e9521e864d43a79d3450570cb82c59 P d113821a62ff45ee8cf6e4ab0ceeb0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d113821a62ff45ee8cf6e4ab0ceeb0b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34251 } } peers { permanent_uuid: "2d958665cb98413cb4da11c123d4b1ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45001 } } peers { permanent_uuid: "5586c77db5354d00ad3eef8f8f4476e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41297 } }
I20260620 00:10:41.467304 18954 ts_tablet_manager.cc:1428] T 36e9521e864d43a79d3450570cb82c59 P d113821a62ff45ee8cf6e4ab0ceeb0b2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 00:10:41.467448 18936 heartbeater.cc:503] Master 127.0.98.62:43457 was elected leader, sending a full tablet report...
I20260620 00:10:41.467492 18956 tablet_bootstrap.cc:492] T 36e9521e864d43a79d3450570cb82c59 P 2d958665cb98413cb4da11c123d4b1ad: No bootstrap required, opened a new log
I20260620 00:10:41.467492 18953 tablet_bootstrap.cc:492] T 36e9521e864d43a79d3450570cb82c59 P 5586c77db5354d00ad3eef8f8f4476e8: No bootstrap required, opened a new log
I20260620 00:10:41.467558 18953 ts_tablet_manager.cc:1397] T 36e9521e864d43a79d3450570cb82c59 P 5586c77db5354d00ad3eef8f8f4476e8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:10:41.467558 18956 ts_tablet_manager.cc:1397] T 36e9521e864d43a79d3450570cb82c59 P 2d958665cb98413cb4da11c123d4b1ad: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:10:41.469606 18953 raft_consensus.cc:348] T 36e9521e864d43a79d3450570cb82c59 P 5586c77db5354d00ad3eef8f8f4476e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d113821a62ff45ee8cf6e4ab0ceeb0b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34251 } } peers { permanent_uuid: "2d958665cb98413cb4da11c123d4b1ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45001 } } peers { permanent_uuid: "5586c77db5354d00ad3eef8f8f4476e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41297 } }
I20260620 00:10:41.469606 18956 raft_consensus.cc:348] T 36e9521e864d43a79d3450570cb82c59 P 2d958665cb98413cb4da11c123d4b1ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d113821a62ff45ee8cf6e4ab0ceeb0b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34251 } } peers { permanent_uuid: "2d958665cb98413cb4da11c123d4b1ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45001 } } peers { permanent_uuid: "5586c77db5354d00ad3eef8f8f4476e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41297 } }
I20260620 00:10:41.469798 18956 raft_consensus.cc:374] T 36e9521e864d43a79d3450570cb82c59 P 2d958665cb98413cb4da11c123d4b1ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:10:41.469798 18953 raft_consensus.cc:374] T 36e9521e864d43a79d3450570cb82c59 P 5586c77db5354d00ad3eef8f8f4476e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:10:41.469836 18953 raft_consensus.cc:729] T 36e9521e864d43a79d3450570cb82c59 P 5586c77db5354d00ad3eef8f8f4476e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5586c77db5354d00ad3eef8f8f4476e8, State: Initialized, Role: FOLLOWER
I20260620 00:10:41.469836 18956 raft_consensus.cc:729] T 36e9521e864d43a79d3450570cb82c59 P 2d958665cb98413cb4da11c123d4b1ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d958665cb98413cb4da11c123d4b1ad, State: Initialized, Role: FOLLOWER
I20260620 00:10:41.469952 18953 consensus_queue.cc:260] T 36e9521e864d43a79d3450570cb82c59 P 5586c77db5354d00ad3eef8f8f4476e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d113821a62ff45ee8cf6e4ab0ceeb0b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34251 } } peers { permanent_uuid: "2d958665cb98413cb4da11c123d4b1ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45001 } } peers { permanent_uuid: "5586c77db5354d00ad3eef8f8f4476e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41297 } }
I20260620 00:10:41.469952 18956 consensus_queue.cc:260] T 36e9521e864d43a79d3450570cb82c59 P 2d958665cb98413cb4da11c123d4b1ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d113821a62ff45ee8cf6e4ab0ceeb0b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34251 } } peers { permanent_uuid: "2d958665cb98413cb4da11c123d4b1ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45001 } } peers { permanent_uuid: "5586c77db5354d00ad3eef8f8f4476e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41297 } }
I20260620 00:10:41.470245 18953 ts_tablet_manager.cc:1428] T 36e9521e864d43a79d3450570cb82c59 P 5586c77db5354d00ad3eef8f8f4476e8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:10:41.470245 18956 ts_tablet_manager.cc:1428] T 36e9521e864d43a79d3450570cb82c59 P 2d958665cb98413cb4da11c123d4b1ad: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:10:41.470285 18802 heartbeater.cc:503] Master 127.0.98.62:43457 was elected leader, sending a full tablet report...
I20260620 00:10:41.470381 18668 heartbeater.cc:503] Master 127.0.98.62:43457 was elected leader, sending a full tablet report...
W20260620 00:10:41.517724 18803 tablet.cc:2369] T 36e9521e864d43a79d3450570cb82c59 P 5586c77db5354d00ad3eef8f8f4476e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 00:10:41.606945 18669 tablet.cc:2369] T 36e9521e864d43a79d3450570cb82c59 P 2d958665cb98413cb4da11c123d4b1ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 00:10:41.674614 18937 tablet.cc:2369] T 36e9521e864d43a79d3450570cb82c59 P d113821a62ff45ee8cf6e4ab0ceeb0b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:10:41.710206 18959 raft_consensus.cc:482] T 36e9521e864d43a79d3450570cb82c59 P d113821a62ff45ee8cf6e4ab0ceeb0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:10:41.710338 18959 raft_consensus.cc:504] T 36e9521e864d43a79d3450570cb82c59 P d113821a62ff45ee8cf6e4ab0ceeb0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d113821a62ff45ee8cf6e4ab0ceeb0b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34251 } } peers { permanent_uuid: "2d958665cb98413cb4da11c123d4b1ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45001 } } peers { permanent_uuid: "5586c77db5354d00ad3eef8f8f4476e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41297 } }
I20260620 00:10:41.710788 18959 leader_election.cc:288] T 36e9521e864d43a79d3450570cb82c59 P d113821a62ff45ee8cf6e4ab0ceeb0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d958665cb98413cb4da11c123d4b1ad (127.0.98.1:45001), 5586c77db5354d00ad3eef8f8f4476e8 (127.0.98.2:41297)
I20260620 00:10:41.714093 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:41.711281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39153 (local address 127.0.98.1:45001)
0620 00:10:41.711354 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:41.711357 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:41.711367 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:10:41.711403 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:41.711404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:41.711418 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:41.711480 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:10:41.711670 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:41.712203 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:41.713210 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:41.713715 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:41.713774 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:41.713783 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:41.713966 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 00:10:41.714013 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 00:10:41.714120 18951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:41.711358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55363 (local address 127.0.98.2:41297)
0620 00:10:41.711440 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:41.711442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:41.711475 (+ 33us) server_negotiation.cc:407] Connection header received
0620 00:10:41.711509 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:41.711510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:41.711522 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:41.711585 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:10:41.711769 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:41.712220 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:41.713330 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:41.713730 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:41.713826 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:41.713835 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:41.714017 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 00:10:41.714060 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 00:10:41.714108 18938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:41.711073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45001 (local address 127.0.98.3:39153)
0620 00:10:41.711142 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:41.711155 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:41.711202 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:41.711503 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:41.711505 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:41.711507 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:10:41.711586 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:41.711588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:41.712296 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:41.712297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:41.713104 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:41.713106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:41.713804 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:41.713805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:41.713894 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:41.713897 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:10:41.713940 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 00:10:41.714016 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 00:10:41.714128 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:41.711204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41297 (local address 127.0.98.3:55363)
0620 00:10:41.711384 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:41.711395 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:41.711462 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:41.711601 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:41.711602 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:41.711603 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:10:41.711693 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:41.711694 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:41.712390 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:41.712391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:41.713235 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:41.713237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:41.713848 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:41.713849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:41.713917 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:41.713920 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:10:41.713997 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 00:10:41.714037 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":104,"threads_started":1}
I20260620 00:10:41.714560 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36e9521e864d43a79d3450570cb82c59" candidate_uuid: "d113821a62ff45ee8cf6e4ab0ceeb0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5586c77db5354d00ad3eef8f8f4476e8" is_pre_election: true
I20260620 00:10:41.714718 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36e9521e864d43a79d3450570cb82c59" candidate_uuid: "d113821a62ff45ee8cf6e4ab0ceeb0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d958665cb98413cb4da11c123d4b1ad" is_pre_election: true
I20260620 00:10:41.714764 18756 raft_consensus.cc:2454] T 36e9521e864d43a79d3450570cb82c59 P 5586c77db5354d00ad3eef8f8f4476e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d113821a62ff45ee8cf6e4ab0ceeb0b2 in term 0.
I20260620 00:10:41.714856 18622 raft_consensus.cc:2454] T 36e9521e864d43a79d3450570cb82c59 P 2d958665cb98413cb4da11c123d4b1ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d113821a62ff45ee8cf6e4ab0ceeb0b2 in term 0.
I20260620 00:10:41.715013 18825 leader_election.cc:302] T 36e9521e864d43a79d3450570cb82c59 P d113821a62ff45ee8cf6e4ab0ceeb0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5586c77db5354d00ad3eef8f8f4476e8, d113821a62ff45ee8cf6e4ab0ceeb0b2; no voters:
I20260620 00:10:41.715150 18959 raft_consensus.cc:2781] T 36e9521e864d43a79d3450570cb82c59 P d113821a62ff45ee8cf6e4ab0ceeb0b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:10:41.715210 18959 raft_consensus.cc:482] T 36e9521e864d43a79d3450570cb82c59 P d113821a62ff45ee8cf6e4ab0ceeb0b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:10:41.715245 18959 raft_consensus.cc:3037] T 36e9521e864d43a79d3450570cb82c59 P d113821a62ff45ee8cf6e4ab0ceeb0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:10:41.716121 18959 raft_consensus.cc:504] T 36e9521e864d43a79d3450570cb82c59 P d113821a62ff45ee8cf6e4ab0ceeb0b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d113821a62ff45ee8cf6e4ab0ceeb0b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34251 } } peers { permanent_uuid: "2d958665cb98413cb4da11c123d4b1ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45001 } } peers { permanent_uuid: "5586c77db5354d00ad3eef8f8f4476e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41297 } }
I20260620 00:10:41.716408 18959 leader_election.cc:288] T 36e9521e864d43a79d3450570cb82c59 P d113821a62ff45ee8cf6e4ab0ceeb0b2 [CANDIDATE]: Term 1 election: Requested vote from peers 2d958665cb98413cb4da11c123d4b1ad (127.0.98.1:45001), 5586c77db5354d00ad3eef8f8f4476e8 (127.0.98.2:41297)
I20260620 00:10:41.716689 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36e9521e864d43a79d3450570cb82c59" candidate_uuid: "d113821a62ff45ee8cf6e4ab0ceeb0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d958665cb98413cb4da11c123d4b1ad"
I20260620 00:10:41.716678 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36e9521e864d43a79d3450570cb82c59" candidate_uuid: "d113821a62ff45ee8cf6e4ab0ceeb0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5586c77db5354d00ad3eef8f8f4476e8"
I20260620 00:10:41.716774 18622 raft_consensus.cc:3037] T 36e9521e864d43a79d3450570cb82c59 P 2d958665cb98413cb4da11c123d4b1ad [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:10:41.716774 18756 raft_consensus.cc:3037] T 36e9521e864d43a79d3450570cb82c59 P 5586c77db5354d00ad3eef8f8f4476e8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:10:41.717729 18622 raft_consensus.cc:2454] T 36e9521e864d43a79d3450570cb82c59 P 2d958665cb98413cb4da11c123d4b1ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d113821a62ff45ee8cf6e4ab0ceeb0b2 in term 1.
I20260620 00:10:41.717729 18756 raft_consensus.cc:2454] T 36e9521e864d43a79d3450570cb82c59 P 5586c77db5354d00ad3eef8f8f4476e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d113821a62ff45ee8cf6e4ab0ceeb0b2 in term 1.
I20260620 00:10:41.717963 18825 leader_election.cc:302] T 36e9521e864d43a79d3450570cb82c59 P d113821a62ff45ee8cf6e4ab0ceeb0b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5586c77db5354d00ad3eef8f8f4476e8, d113821a62ff45ee8cf6e4ab0ceeb0b2; no voters:
I20260620 00:10:41.718066 18959 raft_consensus.cc:2781] T 36e9521e864d43a79d3450570cb82c59 P d113821a62ff45ee8cf6e4ab0ceeb0b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:10:41.718230 18959 raft_consensus.cc:686] T 36e9521e864d43a79d3450570cb82c59 P d113821a62ff45ee8cf6e4ab0ceeb0b2 [term 1 LEADER]: Becoming Leader. State: Replica: d113821a62ff45ee8cf6e4ab0ceeb0b2, State: Running, Role: LEADER
I20260620 00:10:41.718313 18959 consensus_queue.cc:237] T 36e9521e864d43a79d3450570cb82c59 P d113821a62ff45ee8cf6e4ab0ceeb0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d113821a62ff45ee8cf6e4ab0ceeb0b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34251 } } peers { permanent_uuid: "2d958665cb98413cb4da11c123d4b1ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45001 } } peers { permanent_uuid: "5586c77db5354d00ad3eef8f8f4476e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41297 } }
I20260620 00:10:41.719512 18477 catalog_manager.cc:5495] T 36e9521e864d43a79d3450570cb82c59 P d113821a62ff45ee8cf6e4ab0ceeb0b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d113821a62ff45ee8cf6e4ab0ceeb0b2 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d113821a62ff45ee8cf6e4ab0ceeb0b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d113821a62ff45ee8cf6e4ab0ceeb0b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d958665cb98413cb4da11c123d4b1ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5586c77db5354d00ad3eef8f8f4476e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41297 } health_report { overall_health: UNKNOWN } } }
I20260620 00:10:41.743675 18952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:41.741009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37692 (local address 127.0.98.3:34251)
0620 00:10:41.741073 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:41.741074 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:41.741082 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:10:41.741164 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:41.741165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:41.741182 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:41.741243 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:10:41.741357 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:41.741994 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:41.742780 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:41.743020 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:41.743100 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:41.743250 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:10:41.743383 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:10:41.743478 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:41.743526 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 00:10:41.743535 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 00:10:41.745587 18622 raft_consensus.cc:1261] T 36e9521e864d43a79d3450570cb82c59 P 2d958665cb98413cb4da11c123d4b1ad [term 1 FOLLOWER]: Refusing update from remote peer d113821a62ff45ee8cf6e4ab0ceeb0b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:10:41.745607 18756 raft_consensus.cc:1261] T 36e9521e864d43a79d3450570cb82c59 P 5586c77db5354d00ad3eef8f8f4476e8 [term 1 FOLLOWER]: Refusing update from remote peer d113821a62ff45ee8cf6e4ab0ceeb0b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:10:41.746006 18963 consensus_queue.cc:1035] T 36e9521e864d43a79d3450570cb82c59 P d113821a62ff45ee8cf6e4ab0ceeb0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d958665cb98413cb4da11c123d4b1ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:10:41.746152 18959 consensus_queue.cc:1035] T 36e9521e864d43a79d3450570cb82c59 P d113821a62ff45ee8cf6e4ab0ceeb0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5586c77db5354d00ad3eef8f8f4476e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:10:41.748857 18967 mvcc.cc:205] Tried to move back new op lower bound from 7298720734186995712 to 7298720734078787584. Current Snapshot: MvccSnapshot[applied={T|T < 7298720734186995712 or (T in {7298720734186995712})}]
I20260620 00:10:41.749094 18968 mvcc.cc:205] Tried to move back new op lower bound from 7298720734186995712 to 7298720734078787584. Current Snapshot: MvccSnapshot[applied={T|T < 7298720734186995712}]
I20260620 00:10:41.751407 18477 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:10:41.751513 18477 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:10:41.752317 18477 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37858f2a7eda4485b62c4fbd4e9736a0: Sending DeleteTablet for 3 replicas of tablet 36e9521e864d43a79d3450570cb82c59
I20260620 00:10:41.752718 18870 tablet_service.cc:1514] Processing DeleteTablet for tablet 36e9521e864d43a79d3450570cb82c59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:10:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37686
I20260620 00:10:41.752794 18736 tablet_service.cc:1514] Processing DeleteTablet for tablet 36e9521e864d43a79d3450570cb82c59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:10:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39336
I20260620 00:10:41.752974 18976 tablet_replica.cc:330] T 36e9521e864d43a79d3450570cb82c59 P d113821a62ff45ee8cf6e4ab0ceeb0b2: stopping tablet replica
I20260620 00:10:41.753077 18976 raft_consensus.cc:2229] T 36e9521e864d43a79d3450570cb82c59 P d113821a62ff45ee8cf6e4ab0ceeb0b2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:10:41.753075 18602 tablet_service.cc:1514] Processing DeleteTablet for tablet 36e9521e864d43a79d3450570cb82c59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:10:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40382
I20260620 00:10:41.753235 18976 raft_consensus.cc:2258] T 36e9521e864d43a79d3450570cb82c59 P d113821a62ff45ee8cf6e4ab0ceeb0b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:10:41.753330 18978 tablet_replica.cc:330] T 36e9521e864d43a79d3450570cb82c59 P 2d958665cb98413cb4da11c123d4b1ad: stopping tablet replica
I20260620 00:10:41.753504 18978 raft_consensus.cc:2229] T 36e9521e864d43a79d3450570cb82c59 P 2d958665cb98413cb4da11c123d4b1ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:10:41.753674 18978 raft_consensus.cc:2258] T 36e9521e864d43a79d3450570cb82c59 P 2d958665cb98413cb4da11c123d4b1ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:10:41.753722 18976 ts_tablet_manager.cc:1905] T 36e9521e864d43a79d3450570cb82c59 P d113821a62ff45ee8cf6e4ab0ceeb0b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:10:41.753742 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 18539
I20260620 00:10:41.755765 18977 tablet_replica.cc:330] T 36e9521e864d43a79d3450570cb82c59 P 5586c77db5354d00ad3eef8f8f4476e8: stopping tablet replica
I20260620 00:10:41.755888 18977 raft_consensus.cc:2229] T 36e9521e864d43a79d3450570cb82c59 P 5586c77db5354d00ad3eef8f8f4476e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:10:41.755997 18977 raft_consensus.cc:2258] T 36e9521e864d43a79d3450570cb82c59 P 5586c77db5354d00ad3eef8f8f4476e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:10:41.756410 18977 ts_tablet_manager.cc:1905] T 36e9521e864d43a79d3450570cb82c59 P 5586c77db5354d00ad3eef8f8f4476e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:10:41.758430 18977 ts_tablet_manager.cc:1918] T 36e9521e864d43a79d3450570cb82c59 P 5586c77db5354d00ad3eef8f8f4476e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:10:41.758507 18977 log.cc:1196] T 36e9521e864d43a79d3450570cb82c59 P 5586c77db5354d00ad3eef8f8f4476e8: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-1/wal/wals/36e9521e864d43a79d3450570cb82c59
I20260620 00:10:41.758744 18977 ts_tablet_manager.cc:1939] T 36e9521e864d43a79d3450570cb82c59 P 5586c77db5354d00ad3eef8f8f4476e8: Deleting consensus metadata
I20260620 00:10:41.759315 18463 catalog_manager.cc:4841] TS 5586c77db5354d00ad3eef8f8f4476e8 (127.0.98.2:41297): tablet 36e9521e864d43a79d3450570cb82c59 (table test-table [id=b24c9375b1f24cf1bce023f812ecae72]) successfully deleted
W20260620 00:10:41.760010 18463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45001 (error 108)
W20260620 00:10:41.761485 18947 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:10:41.760415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45001 (local address 127.0.0.1:40396)
0620 00:10:41.760464 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:41.760476 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:41.760513 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:41.761428 (+ 915us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45001: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 00:10:41.761607 18463 catalog_manager.cc:4584] TS 2d958665cb98413cb4da11c123d4b1ad (127.0.98.1:45001): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 36e9521e864d43a79d3450570cb82c59: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45001: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 00:10:41.761781 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 18673
I20260620 00:10:41.761796 18976 ts_tablet_manager.cc:1918] T 36e9521e864d43a79d3450570cb82c59 P d113821a62ff45ee8cf6e4ab0ceeb0b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:10:41.761934 18976 log.cc:1196] T 36e9521e864d43a79d3450570cb82c59 P d113821a62ff45ee8cf6e4ab0ceeb0b2: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914001714546-392-0/minicluster-data/ts-2/wal/wals/36e9521e864d43a79d3450570cb82c59
I20260620 00:10:41.762260 18976 ts_tablet_manager.cc:1939] T 36e9521e864d43a79d3450570cb82c59 P d113821a62ff45ee8cf6e4ab0ceeb0b2: Deleting consensus metadata
I20260620 00:10:41.762689 18461 catalog_manager.cc:4841] TS d113821a62ff45ee8cf6e4ab0ceeb0b2 (127.0.98.3:34251): tablet 36e9521e864d43a79d3450570cb82c59 (table test-table [id=b24c9375b1f24cf1bce023f812ecae72]) successfully deleted
I20260620 00:10:41.767792 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 18807
I20260620 00:10:41.772780 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 18445
2026-06-20T00:10:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2152 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T00:10:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:10:41Z Disabled control of system clock
I20260620 00:10:41.801077 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39813
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 00:10:41.882902 18983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:10:41.883179 18983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:10:41.883203 18983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:10:41.884471 18983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:10:41.884523 18983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:10:41.884536 18983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:10:41.884549 18983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:10:41.884562 18983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:10:41.886039 18983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39813
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:10:41.886320 18983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:10:41.886657 18983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:10:41.891748 18989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:41.891893 18983 server_base.cc:1034] running on GCE node
W20260620 00:10:41.891748 18991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:41.892058 18988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:41.892330 18983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:10:41.892658 18983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:10:41.893841 18983 hybrid_clock.cc:648] HybridClock initialized: now 1781914241893833 us; error 36 us; skew 500 ppm
I20260620 00:10:41.895236 18983 webserver.cc:460] Webserver started at http://127.0.98.62:44535/ using document root <none> and password file <none>
I20260620 00:10:41.895537 18983 fs_manager.cc:359] Metadata directory not provided
I20260620 00:10:41.895586 18983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:10:41.895737 18983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:10:41.896844 18983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "22d5b4e9ea0f416f993f5ef87cef709d"
format_stamp: "Formatted at 2026-06-20 00:10:41 on dist-test-slave-grqt"
server_key: "59a180339d23d7511982081062047d7d"
server_key_iv: "9396a1a5069a8cef0298803da20f25a8"
server_key_version: "encryptionkey@0"
I20260620 00:10:41.897228 18983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "22d5b4e9ea0f416f993f5ef87cef709d"
format_stamp: "Formatted at 2026-06-20 00:10:41 on dist-test-slave-grqt"
server_key: "59a180339d23d7511982081062047d7d"
server_key_iv: "9396a1a5069a8cef0298803da20f25a8"
server_key_version: "encryptionkey@0"
I20260620 00:10:41.898610 18983 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:10:41.899495 18997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:41.899756 18983 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:10:41.899843 18983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "22d5b4e9ea0f416f993f5ef87cef709d"
format_stamp: "Formatted at 2026-06-20 00:10:41 on dist-test-slave-grqt"
server_key: "59a180339d23d7511982081062047d7d"
server_key_iv: "9396a1a5069a8cef0298803da20f25a8"
server_key_version: "encryptionkey@0"
I20260620 00:10:41.899960 18983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:10:41.927699 18983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:10:41.928074 18983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:10:41.928246 18983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:10:41.932447 18983 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39813
I20260620 00:10:41.932487 19049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39813 every 8 connection(s)
I20260620 00:10:41.932835 18983 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:10:41.934094 19050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:10:41.935971 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 18983
I20260620 00:10:41.936058 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:10:41.936267 392 external_mini_cluster.cc:1428] Setting key 738baa19b709fd7b33a8223a482e5757
I20260620 00:10:41.936894 19050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22d5b4e9ea0f416f993f5ef87cef709d: Bootstrap starting.
I20260620 00:10:41.937635 19050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22d5b4e9ea0f416f993f5ef87cef709d: Neither blocks nor log segments found. Creating new log.
I20260620 00:10:41.937953 19050 log.cc:824] T 00000000000000000000000000000000 P 22d5b4e9ea0f416f993f5ef87cef709d: Log is configured to *not* fsync() on all Append() calls
I20260620 00:10:41.938683 19050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22d5b4e9ea0f416f993f5ef87cef709d: No bootstrap required, opened a new log
I20260620 00:10:41.940914 19050 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22d5b4e9ea0f416f993f5ef87cef709d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22d5b4e9ea0f416f993f5ef87cef709d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } }
I20260620 00:10:41.941114 19050 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22d5b4e9ea0f416f993f5ef87cef709d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:10:41.941156 19050 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22d5b4e9ea0f416f993f5ef87cef709d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22d5b4e9ea0f416f993f5ef87cef709d, State: Initialized, Role: FOLLOWER
I20260620 00:10:41.941288 19050 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22d5b4e9ea0f416f993f5ef87cef709d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22d5b4e9ea0f416f993f5ef87cef709d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } }
I20260620 00:10:41.941344 19050 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22d5b4e9ea0f416f993f5ef87cef709d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:10:41.941411 19050 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22d5b4e9ea0f416f993f5ef87cef709d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:10:41.941457 19050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22d5b4e9ea0f416f993f5ef87cef709d [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:10:41.942147 19050 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22d5b4e9ea0f416f993f5ef87cef709d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22d5b4e9ea0f416f993f5ef87cef709d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } }
I20260620 00:10:41.942260 19050 leader_election.cc:302] T 00000000000000000000000000000000 P 22d5b4e9ea0f416f993f5ef87cef709d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22d5b4e9ea0f416f993f5ef87cef709d; no voters:
I20260620 00:10:41.942436 19053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:41.936805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56930 (local address 127.0.98.62:39813)
0620 00:10:41.937022 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:41.937031 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:41.937063 (+ 32us) server_negotiation.cc:407] Connection header received
0620 00:10:41.937672 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:41.937681 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:41.937766 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:41.937923 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:41.938292 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:41.939287 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:41.940415 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:41.940814 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:41.940903 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:41.941715 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:41.941718 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:41.941718 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:10:41.941880 (+ 162us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:41.941926 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:10:41.941937 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:41.941987 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:41.942097 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 00:10:41.942214 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":81,"threads_started":1}
I20260620 00:10:41.942469 19050 leader_election.cc:288] T 00000000000000000000000000000000 P 22d5b4e9ea0f416f993f5ef87cef709d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:10:41.942600 19055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22d5b4e9ea0f416f993f5ef87cef709d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:10:41.942723 19055 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22d5b4e9ea0f416f993f5ef87cef709d [term 1 LEADER]: Becoming Leader. State: Replica: 22d5b4e9ea0f416f993f5ef87cef709d, State: Running, Role: LEADER
I20260620 00:10:41.942741 19050 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22d5b4e9ea0f416f993f5ef87cef709d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:10:41.942809 19055 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22d5b4e9ea0f416f993f5ef87cef709d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22d5b4e9ea0f416f993f5ef87cef709d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } }
I20260620 00:10:41.943149 19055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22d5b4e9ea0f416f993f5ef87cef709d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22d5b4e9ea0f416f993f5ef87cef709d. Latest consensus state: current_term: 1 leader_uuid: "22d5b4e9ea0f416f993f5ef87cef709d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22d5b4e9ea0f416f993f5ef87cef709d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } }
I20260620 00:10:41.943250 19055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22d5b4e9ea0f416f993f5ef87cef709d [sys.catalog]: This master's current role is: LEADER
I20260620 00:10:41.943403 19056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22d5b4e9ea0f416f993f5ef87cef709d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22d5b4e9ea0f416f993f5ef87cef709d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22d5b4e9ea0f416f993f5ef87cef709d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } }
I20260620 00:10:41.943477 19056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22d5b4e9ea0f416f993f5ef87cef709d [sys.catalog]: This master's current role is: LEADER
I20260620 00:10:41.943456 19063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:10:41.944072 19063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:10:41.945683 19063 catalog_manager.cc:1347] Generated new cluster ID: 9a8ec990bbc1407ab80824f29300665e
I20260620 00:10:41.945739 19063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:10:41.953543 19063 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:10:41.954205 19063 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:10:41.964466 19063 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22d5b4e9ea0f416f993f5ef87cef709d: Generated new TSK 0
I20260620 00:10:41.964679 19063 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:10:41.967738 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39813
--builtin_ntp_servers=127.0.98.20:40017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 00:10:42.056746 19074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:10:42.056942 19074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:10:42.056963 19074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:10:42.058144 19074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:10:42.058185 19074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:10:42.058211 19074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:10:42.059552 19074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:10:42.059792 19074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:10:42.060111 19074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:10:42.065222 19079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:42.065259 19080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:42.065455 19074 server_base.cc:1034] running on GCE node
W20260620 00:10:42.065233 19082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:42.065820 19074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:10:42.066102 19074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:10:42.067304 19074 hybrid_clock.cc:648] HybridClock initialized: now 1781914242067290 us; error 28 us; skew 500 ppm
I20260620 00:10:42.071957 19074 webserver.cc:460] Webserver started at http://127.0.98.1:38689/ using document root <none> and password file <none>
I20260620 00:10:42.072312 19074 fs_manager.cc:359] Metadata directory not provided
I20260620 00:10:42.072387 19074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:10:42.072546 19074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:10:42.074189 19074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-0/data/instance:
uuid: "4f23e37fdaec4127977924bc37348346"
format_stamp: "Formatted at 2026-06-20 00:10:42 on dist-test-slave-grqt"
server_key: "6b309da4e61d99b88e222f47fefc8819"
server_key_iv: "dc73948d1c0508fc19ba21f85df4fa27"
server_key_version: "encryptionkey@0"
I20260620 00:10:42.074652 19074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4f23e37fdaec4127977924bc37348346"
format_stamp: "Formatted at 2026-06-20 00:10:42 on dist-test-slave-grqt"
server_key: "6b309da4e61d99b88e222f47fefc8819"
server_key_iv: "dc73948d1c0508fc19ba21f85df4fa27"
server_key_version: "encryptionkey@0"
I20260620 00:10:42.076547 19074 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 00:10:42.077616 19088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:42.077798 19074 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:10:42.077880 19074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-0/wal
uuid: "4f23e37fdaec4127977924bc37348346"
format_stamp: "Formatted at 2026-06-20 00:10:42 on dist-test-slave-grqt"
server_key: "6b309da4e61d99b88e222f47fefc8819"
server_key_iv: "dc73948d1c0508fc19ba21f85df4fa27"
server_key_version: "encryptionkey@0"
I20260620 00:10:42.077950 19074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:10:42.098855 19074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:10:42.099239 19074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:10:42.099409 19074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:10:42.099682 19074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:10:42.100085 19074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:10:42.100150 19074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:42.100198 19074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:10:42.100230 19074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:42.106694 19074 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45321
I20260620 00:10:42.106766 19201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45321 every 8 connection(s)
I20260620 00:10:42.107165 19074 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:10:42.113058 19053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:42.108022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58165 (local address 127.0.98.62:39813)
0620 00:10:42.108096 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:42.108099 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:42.108368 (+ 269us) server_negotiation.cc:407] Connection header received
0620 00:10:42.108936 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:42.108937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:42.108952 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:42.109016 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:42.109861 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:42.110399 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:42.111517 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:42.111716 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:42.111780 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:42.112791 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:42.112792 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:42.112792 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:10:42.112796 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:42.112810 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:10:42.112814 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:42.112860 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:42.112946 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 00:10:42.112996 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 00:10:42.113260 19204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:42.108001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.98.1:58165)
0620 00:10:42.108238 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:42.108278 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:42.108775 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:42.109121 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:42.109124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:42.109132 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:10:42.109747 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:42.109753 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:42.110491 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:42.110491 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:42.111295 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:42.111297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:42.111826 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:42.111827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:42.111958 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:42.112626 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:10:42.112627 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:10:42.112659 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:10:42.112683 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:10:42.112878 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:10:42.112881 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:10:42.112884 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:10:42.112936 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 00:10:42.113033 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":116,"threads_started":1}
I20260620 00:10:42.113864 19202 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39813
I20260620 00:10:42.113978 19202 heartbeater.cc:464] Registering TS with master...
I20260620 00:10:42.114176 19202 heartbeater.cc:511] Master 127.0.98.62:39813 requested a full tablet report, sending...
I20260620 00:10:42.114802 19014 ts_manager.cc:194] Registered new tserver with Master: 4f23e37fdaec4127977924bc37348346 (127.0.98.1:45321)
I20260620 00:10:42.115471 19014 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:58165
I20260620 00:10:42.116606 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 19074
I20260620 00:10:42.116721 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:10:42.116950 392 external_mini_cluster.cc:1428] Setting key 411ab78ecc37b392a408056dd4d6a233
I20260620 00:10:42.118023 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39813
--builtin_ntp_servers=127.0.98.20:40017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 00:10:42.201498 19205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:10:42.201712 19205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:10:42.201737 19205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:10:42.202967 19205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:10:42.203050 19205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:10:42.203117 19205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:10:42.204491 19205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:10:42.204694 19205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:10:42.205014 19205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:10:42.210263 19211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:42.210271 19210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:42.210407 19205 server_base.cc:1034] running on GCE node
W20260620 00:10:42.210271 19213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:42.210793 19205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:10:42.211097 19205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:10:42.212291 19205 hybrid_clock.cc:648] HybridClock initialized: now 1781914242212267 us; error 37 us; skew 500 ppm
I20260620 00:10:42.213718 19205 webserver.cc:460] Webserver started at http://127.0.98.2:39177/ using document root <none> and password file <none>
I20260620 00:10:42.213984 19205 fs_manager.cc:359] Metadata directory not provided
I20260620 00:10:42.214063 19205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:10:42.214228 19205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:10:42.215453 19205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-1/data/instance:
uuid: "7ab50bda535e4d07bf27344ef723479a"
format_stamp: "Formatted at 2026-06-20 00:10:42 on dist-test-slave-grqt"
server_key: "5c7785d12ab4adab5802299fd2d48d82"
server_key_iv: "d5252a6fcdc2fe4f6b6bb00044f27b84"
server_key_version: "encryptionkey@0"
I20260620 00:10:42.215862 19205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7ab50bda535e4d07bf27344ef723479a"
format_stamp: "Formatted at 2026-06-20 00:10:42 on dist-test-slave-grqt"
server_key: "5c7785d12ab4adab5802299fd2d48d82"
server_key_iv: "d5252a6fcdc2fe4f6b6bb00044f27b84"
server_key_version: "encryptionkey@0"
I20260620 00:10:42.217286 19205 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:10:42.218169 19219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:42.218362 19205 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:10:42.218479 19205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-1/wal
uuid: "7ab50bda535e4d07bf27344ef723479a"
format_stamp: "Formatted at 2026-06-20 00:10:42 on dist-test-slave-grqt"
server_key: "5c7785d12ab4adab5802299fd2d48d82"
server_key_iv: "d5252a6fcdc2fe4f6b6bb00044f27b84"
server_key_version: "encryptionkey@0"
I20260620 00:10:42.218568 19205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:10:42.230521 19205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:10:42.230850 19205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:10:42.231012 19205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:10:42.231253 19205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:10:42.231611 19205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:10:42.231671 19205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:42.231716 19205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:10:42.231748 19205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:42.238129 19205 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36119
I20260620 00:10:42.238188 19332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36119 every 8 connection(s)
I20260620 00:10:42.238576 19205 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:10:42.242846 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 19205
I20260620 00:10:42.242969 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:10:42.243234 392 external_mini_cluster.cc:1428] Setting key 765daffb009e8781722803b5f8fea7a8
I20260620 00:10:42.244313 19053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:42.239318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56119 (local address 127.0.98.62:39813)
0620 00:10:42.239362 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:42.239364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:42.239582 (+ 218us) server_negotiation.cc:407] Connection header received
0620 00:10:42.240059 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:42.240060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:42.240071 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:42.240136 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:42.240854 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:42.241489 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:42.242596 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:42.242982 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:42.243082 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:42.244039 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:42.244040 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:42.244040 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:10:42.244046 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:42.244062 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:10:42.244066 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:42.244133 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:42.244236 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 00:10:42.244257 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 00:10:42.244417 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39813
--builtin_ntp_servers=127.0.98.20:40017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 00:10:42.244696 19335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:42.239261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.98.2:56119)
0620 00:10:42.239439 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:42.239479 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:42.239960 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:42.240205 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:42.240208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:42.240216 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:10:42.240766 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:42.240771 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:42.241578 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:42.241579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:42.242477 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:42.242479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:42.243094 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:42.243095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:42.243212 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:42.243844 (+ 632us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:10:42.243845 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:10:42.243903 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:10:42.243936 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:10:42.244141 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:10:42.244148 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:10:42.244151 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:10:42.244241 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 00:10:42.244412 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":92,"threads_started":1}
I20260620 00:10:42.245767 19333 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39813
I20260620 00:10:42.245882 19333 heartbeater.cc:464] Registering TS with master...
I20260620 00:10:42.246142 19333 heartbeater.cc:511] Master 127.0.98.62:39813 requested a full tablet report, sending...
I20260620 00:10:42.246524 19014 ts_manager.cc:194] Registered new tserver with Master: 7ab50bda535e4d07bf27344ef723479a (127.0.98.2:36119)
I20260620 00:10:42.246929 19014 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:56119
W20260620 00:10:42.326133 19336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:10:42.326377 19336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:10:42.326416 19336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:10:42.327590 19336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:10:42.327657 19336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:10:42.327701 19336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:10:42.329089 19336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:10:42.329380 19336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:10:42.329694 19336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:10:42.334653 19342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:42.334674 19336 server_base.cc:1034] running on GCE node
W20260620 00:10:42.334715 19344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:42.334657 19341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:42.334977 19336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:10:42.335191 19336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:10:42.336370 19336 hybrid_clock.cc:648] HybridClock initialized: now 1781914242336358 us; error 29 us; skew 500 ppm
I20260620 00:10:42.337664 19336 webserver.cc:460] Webserver started at http://127.0.98.3:38967/ using document root <none> and password file <none>
I20260620 00:10:42.337904 19336 fs_manager.cc:359] Metadata directory not provided
I20260620 00:10:42.337975 19336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:10:42.338111 19336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:10:42.339165 19336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-2/data/instance:
uuid: "c87d0d97f45c42ce88521903c84838de"
format_stamp: "Formatted at 2026-06-20 00:10:42 on dist-test-slave-grqt"
server_key: "ddd53df79bdc47af60acca3329d195c1"
server_key_iv: "a14140960fb2ef22290b41e3fe554411"
server_key_version: "encryptionkey@0"
I20260620 00:10:42.339538 19336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c87d0d97f45c42ce88521903c84838de"
format_stamp: "Formatted at 2026-06-20 00:10:42 on dist-test-slave-grqt"
server_key: "ddd53df79bdc47af60acca3329d195c1"
server_key_iv: "a14140960fb2ef22290b41e3fe554411"
server_key_version: "encryptionkey@0"
I20260620 00:10:42.340809 19336 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:10:42.341566 19350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:42.341734 19336 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:10:42.341835 19336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-2/wal
uuid: "c87d0d97f45c42ce88521903c84838de"
format_stamp: "Formatted at 2026-06-20 00:10:42 on dist-test-slave-grqt"
server_key: "ddd53df79bdc47af60acca3329d195c1"
server_key_iv: "a14140960fb2ef22290b41e3fe554411"
server_key_version: "encryptionkey@0"
I20260620 00:10:42.341922 19336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:10:42.357303 19336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:10:42.357633 19336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:10:42.357784 19336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:10:42.358006 19336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:10:42.358325 19336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:10:42.358381 19336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:42.358421 19336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:10:42.358449 19336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:42.364148 19336 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35151
I20260620 00:10:42.364187 19463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35151 every 8 connection(s)
I20260620 00:10:42.364576 19336 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:10:42.369513 19053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:42.365309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58081 (local address 127.0.98.62:39813)
0620 00:10:42.365411 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:42.365414 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:42.365614 (+ 200us) server_negotiation.cc:407] Connection header received
0620 00:10:42.366076 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:42.366077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:42.366088 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:42.366141 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:42.366781 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:42.367266 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:42.368209 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:42.368386 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:42.368446 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:42.369196 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:42.369198 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:42.369198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:10:42.369202 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:42.369216 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:10:42.369220 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:42.369271 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:42.369337 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 00:10:42.369359 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 00:10:42.369643 19466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:42.365289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.98.3:58081)
0620 00:10:42.365497 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:42.365529 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:42.365992 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:42.366192 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:42.366195 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:42.366202 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:10:42.366684 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:42.366688 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:42.367350 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:42.367351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:42.368132 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:42.368134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:42.368459 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:42.368459 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:42.368562 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:42.369067 (+ 505us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:10:42.369067 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:10:42.369094 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:10:42.369114 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:10:42.369274 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:10:42.369275 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:10:42.369278 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:10:42.369324 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 00:10:42.369458 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":115,"threads_started":1}
I20260620 00:10:42.370040 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 19336
I20260620 00:10:42.370065 19464 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39813
I20260620 00:10:42.370154 19464 heartbeater.cc:464] Registering TS with master...
I20260620 00:10:42.370157 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:10:42.370353 19464 heartbeater.cc:511] Master 127.0.98.62:39813 requested a full tablet report, sending...
I20260620 00:10:42.370369 392 external_mini_cluster.cc:1428] Setting key f7ff17ddb1f66d854a86e01903fbbfeb
I20260620 00:10:42.370738 19014 ts_manager.cc:194] Registered new tserver with Master: c87d0d97f45c42ce88521903c84838de (127.0.98.3:35151)
I20260620 00:10:42.371233 19014 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:58081
I20260620 00:10:42.371407 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:10:42.376380 19053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:42.373020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56940 (local address 127.0.98.62:39813)
0620 00:10:42.373066 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:42.373068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:42.373572 (+ 504us) server_negotiation.cc:407] Connection header received
0620 00:10:42.373594 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:42.373594 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:42.373603 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:42.373684 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:42.373871 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:42.374338 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:42.375081 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:42.375287 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:42.375360 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:42.376068 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:42.376069 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:42.376069 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:10:42.376074 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:42.376087 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:10:42.376091 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:42.376200 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:42.376266 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 00:10:42.376276 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 00:10:42.378006 19014 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:10:42.380019 19014 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:10:42.386246 19474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:42.382177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45321 (local address 127.0.0.1:38322)
0620 00:10:42.382622 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:42.382651 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:42.382741 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:42.383028 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:42.383031 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:42.383033 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:10:42.383099 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:42.383101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:42.384444 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:42.384444 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:42.385208 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:42.385210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:42.385948 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:42.385948 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:42.386028 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:42.386042 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 00:10:42.386111 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 00:10:42.386163 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":20,"thread_start_us":52,"threads_started":1}
I20260620 00:10:42.386261 19475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:42.382271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38322 (local address 127.0.98.1:45321)
0620 00:10:42.382415 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:42.382419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:42.382723 (+ 304us) server_negotiation.cc:407] Connection header received
0620 00:10:42.382827 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:42.382829 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:42.382842 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:42.382933 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:10:42.383180 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:42.384345 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:42.385302 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:42.385839 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:42.385946 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:42.386042 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:42.386137 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 00:10:42.386177 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":73,"threads_started":1}
I20260620 00:10:42.386711 19472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:42.382180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36119 (local address 127.0.0.1:35550)
0620 00:10:42.382989 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:42.382997 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:42.383041 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:42.383295 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:42.383296 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:42.383297 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:10:42.383328 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:42.383329 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:42.384200 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:42.384201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:42.385431 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:42.385434 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:42.386445 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:42.386445 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:42.386509 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:42.386512 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:10:42.386591 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 00:10:42.386630 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"mutex_wait_us":3,"spinlock_wait_cycles":768,"thread_start_us":95,"threads_started":1}
I20260620 00:10:42.386766 19476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:42.382404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35550 (local address 127.0.98.2:36119)
0620 00:10:42.382618 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:42.382622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:42.383096 (+ 474us) server_negotiation.cc:407] Connection header received
0620 00:10:42.383140 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:42.383142 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:42.383159 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:42.383227 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:10:42.383397 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:42.384112 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:42.385526 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:42.386259 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:42.386323 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:42.386412 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:42.386610 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 00:10:42.386646 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":117,"threads_started":1}
I20260620 00:10:42.387107 19473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:42.382180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35151 (local address 127.0.0.1:52132)
0620 00:10:42.382831 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:42.382838 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:42.382910 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:42.383125 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:42.383126 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:42.383128 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:10:42.383161 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:42.383163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:42.384431 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:42.384431 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:42.385833 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:42.385835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:42.386866 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:42.386867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:42.386928 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:42.386931 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:10:42.386999 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 00:10:42.387037 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"mutex_wait_us":26,"thread_start_us":100,"threads_started":1}
I20260620 00:10:42.387154 19477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:42.382436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52132 (local address 127.0.98.3:35151)
0620 00:10:42.382637 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:42.382640 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:42.382905 (+ 265us) server_negotiation.cc:407] Connection header received
0620 00:10:42.382974 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:42.382976 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:42.382990 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:42.383065 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:10:42.383235 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:42.384355 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:42.385944 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:42.386682 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:42.386751 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:42.386832 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:42.387027 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 00:10:42.387071 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":105,"threads_started":1}
I20260620 00:10:42.387245 19136 tablet_service.cc:1467] Processing CreateTablet for tablet 94ab22af3cc448f69f823d2a99026198 (DEFAULT_TABLE table=test-table [id=4414f1acbbed4341b4be09483b57df13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:10:42.387388 19267 tablet_service.cc:1467] Processing CreateTablet for tablet 94ab22af3cc448f69f823d2a99026198 (DEFAULT_TABLE table=test-table [id=4414f1acbbed4341b4be09483b57df13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:10:42.387555 19136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94ab22af3cc448f69f823d2a99026198. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:10:42.387632 19267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94ab22af3cc448f69f823d2a99026198. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:10:42.390120 19478 tablet_bootstrap.cc:492] T 94ab22af3cc448f69f823d2a99026198 P 4f23e37fdaec4127977924bc37348346: Bootstrap starting.
I20260620 00:10:42.390092 19398 tablet_service.cc:1467] Processing CreateTablet for tablet 94ab22af3cc448f69f823d2a99026198 (DEFAULT_TABLE table=test-table [id=4414f1acbbed4341b4be09483b57df13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:10:42.390437 19398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94ab22af3cc448f69f823d2a99026198. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:10:42.390674 19479 tablet_bootstrap.cc:492] T 94ab22af3cc448f69f823d2a99026198 P 7ab50bda535e4d07bf27344ef723479a: Bootstrap starting.
I20260620 00:10:42.390801 19478 tablet_bootstrap.cc:654] T 94ab22af3cc448f69f823d2a99026198 P 4f23e37fdaec4127977924bc37348346: Neither blocks nor log segments found. Creating new log.
I20260620 00:10:42.391093 19478 log.cc:824] T 94ab22af3cc448f69f823d2a99026198 P 4f23e37fdaec4127977924bc37348346: Log is configured to *not* fsync() on all Append() calls
I20260620 00:10:42.391294 19479 tablet_bootstrap.cc:654] T 94ab22af3cc448f69f823d2a99026198 P 7ab50bda535e4d07bf27344ef723479a: Neither blocks nor log segments found. Creating new log.
I20260620 00:10:42.391619 19479 log.cc:824] T 94ab22af3cc448f69f823d2a99026198 P 7ab50bda535e4d07bf27344ef723479a: Log is configured to *not* fsync() on all Append() calls
I20260620 00:10:42.391906 19478 tablet_bootstrap.cc:492] T 94ab22af3cc448f69f823d2a99026198 P 4f23e37fdaec4127977924bc37348346: No bootstrap required, opened a new log
I20260620 00:10:42.391984 19478 ts_tablet_manager.cc:1397] T 94ab22af3cc448f69f823d2a99026198 P 4f23e37fdaec4127977924bc37348346: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:10:42.392361 19479 tablet_bootstrap.cc:492] T 94ab22af3cc448f69f823d2a99026198 P 7ab50bda535e4d07bf27344ef723479a: No bootstrap required, opened a new log
I20260620 00:10:42.392500 19479 ts_tablet_manager.cc:1397] T 94ab22af3cc448f69f823d2a99026198 P 7ab50bda535e4d07bf27344ef723479a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:10:42.393877 19482 tablet_bootstrap.cc:492] T 94ab22af3cc448f69f823d2a99026198 P c87d0d97f45c42ce88521903c84838de: Bootstrap starting.
I20260620 00:10:42.394289 19478 raft_consensus.cc:348] T 94ab22af3cc448f69f823d2a99026198 P 4f23e37fdaec4127977924bc37348346 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f23e37fdaec4127977924bc37348346" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45321 } } peers { permanent_uuid: "7ab50bda535e4d07bf27344ef723479a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36119 } } peers { permanent_uuid: "c87d0d97f45c42ce88521903c84838de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35151 } }
I20260620 00:10:42.394464 19478 raft_consensus.cc:374] T 94ab22af3cc448f69f823d2a99026198 P 4f23e37fdaec4127977924bc37348346 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:10:42.394506 19482 tablet_bootstrap.cc:654] T 94ab22af3cc448f69f823d2a99026198 P c87d0d97f45c42ce88521903c84838de: Neither blocks nor log segments found. Creating new log.
I20260620 00:10:42.394526 19478 raft_consensus.cc:729] T 94ab22af3cc448f69f823d2a99026198 P 4f23e37fdaec4127977924bc37348346 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f23e37fdaec4127977924bc37348346, State: Initialized, Role: FOLLOWER
I20260620 00:10:42.394647 19478 consensus_queue.cc:260] T 94ab22af3cc448f69f823d2a99026198 P 4f23e37fdaec4127977924bc37348346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f23e37fdaec4127977924bc37348346" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45321 } } peers { permanent_uuid: "7ab50bda535e4d07bf27344ef723479a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36119 } } peers { permanent_uuid: "c87d0d97f45c42ce88521903c84838de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35151 } }
I20260620 00:10:42.394840 19482 log.cc:824] T 94ab22af3cc448f69f823d2a99026198 P c87d0d97f45c42ce88521903c84838de: Log is configured to *not* fsync() on all Append() calls
I20260620 00:10:42.394920 19478 ts_tablet_manager.cc:1428] T 94ab22af3cc448f69f823d2a99026198 P 4f23e37fdaec4127977924bc37348346: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:10:42.394955 19479 raft_consensus.cc:348] T 94ab22af3cc448f69f823d2a99026198 P 7ab50bda535e4d07bf27344ef723479a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f23e37fdaec4127977924bc37348346" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45321 } } peers { permanent_uuid: "7ab50bda535e4d07bf27344ef723479a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36119 } } peers { permanent_uuid: "c87d0d97f45c42ce88521903c84838de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35151 } }
I20260620 00:10:42.395099 19202 heartbeater.cc:503] Master 127.0.98.62:39813 was elected leader, sending a full tablet report...
I20260620 00:10:42.395118 19479 raft_consensus.cc:374] T 94ab22af3cc448f69f823d2a99026198 P 7ab50bda535e4d07bf27344ef723479a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:10:42.395169 19479 raft_consensus.cc:729] T 94ab22af3cc448f69f823d2a99026198 P 7ab50bda535e4d07bf27344ef723479a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ab50bda535e4d07bf27344ef723479a, State: Initialized, Role: FOLLOWER
I20260620 00:10:42.395401 19479 consensus_queue.cc:260] T 94ab22af3cc448f69f823d2a99026198 P 7ab50bda535e4d07bf27344ef723479a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f23e37fdaec4127977924bc37348346" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45321 } } peers { permanent_uuid: "7ab50bda535e4d07bf27344ef723479a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36119 } } peers { permanent_uuid: "c87d0d97f45c42ce88521903c84838de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35151 } }
I20260620 00:10:42.395635 19482 tablet_bootstrap.cc:492] T 94ab22af3cc448f69f823d2a99026198 P c87d0d97f45c42ce88521903c84838de: No bootstrap required, opened a new log
I20260620 00:10:42.395684 19479 ts_tablet_manager.cc:1428] T 94ab22af3cc448f69f823d2a99026198 P 7ab50bda535e4d07bf27344ef723479a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:10:42.395702 19333 heartbeater.cc:503] Master 127.0.98.62:39813 was elected leader, sending a full tablet report...
I20260620 00:10:42.395803 19482 ts_tablet_manager.cc:1397] T 94ab22af3cc448f69f823d2a99026198 P c87d0d97f45c42ce88521903c84838de: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:10:42.397873 19482 raft_consensus.cc:348] T 94ab22af3cc448f69f823d2a99026198 P c87d0d97f45c42ce88521903c84838de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f23e37fdaec4127977924bc37348346" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45321 } } peers { permanent_uuid: "7ab50bda535e4d07bf27344ef723479a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36119 } } peers { permanent_uuid: "c87d0d97f45c42ce88521903c84838de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35151 } }
I20260620 00:10:42.398044 19482 raft_consensus.cc:374] T 94ab22af3cc448f69f823d2a99026198 P c87d0d97f45c42ce88521903c84838de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:10:42.398072 19482 raft_consensus.cc:729] T 94ab22af3cc448f69f823d2a99026198 P c87d0d97f45c42ce88521903c84838de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87d0d97f45c42ce88521903c84838de, State: Initialized, Role: FOLLOWER
I20260620 00:10:42.398212 19482 consensus_queue.cc:260] T 94ab22af3cc448f69f823d2a99026198 P c87d0d97f45c42ce88521903c84838de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f23e37fdaec4127977924bc37348346" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45321 } } peers { permanent_uuid: "7ab50bda535e4d07bf27344ef723479a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36119 } } peers { permanent_uuid: "c87d0d97f45c42ce88521903c84838de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35151 } }
I20260620 00:10:42.398451 19482 ts_tablet_manager.cc:1428] T 94ab22af3cc448f69f823d2a99026198 P c87d0d97f45c42ce88521903c84838de: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 00:10:42.398525 19464 heartbeater.cc:503] Master 127.0.98.62:39813 was elected leader, sending a full tablet report...
W20260620 00:10:42.489193 19334 tablet.cc:2369] T 94ab22af3cc448f69f823d2a99026198 P 7ab50bda535e4d07bf27344ef723479a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:10:42.574069 19486 raft_consensus.cc:482] T 94ab22af3cc448f69f823d2a99026198 P c87d0d97f45c42ce88521903c84838de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:10:42.574213 19486 raft_consensus.cc:504] T 94ab22af3cc448f69f823d2a99026198 P c87d0d97f45c42ce88521903c84838de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f23e37fdaec4127977924bc37348346" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45321 } } peers { permanent_uuid: "7ab50bda535e4d07bf27344ef723479a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36119 } } peers { permanent_uuid: "c87d0d97f45c42ce88521903c84838de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35151 } }
I20260620 00:10:42.574553 19486 leader_election.cc:288] T 94ab22af3cc448f69f823d2a99026198 P c87d0d97f45c42ce88521903c84838de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f23e37fdaec4127977924bc37348346 (127.0.98.1:45321), 7ab50bda535e4d07bf27344ef723479a (127.0.98.2:36119)
I20260620 00:10:42.577303 19466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:42.574752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36119 (local address 127.0.98.3:42687)
0620 00:10:42.574810 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:42.574823 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:42.574865 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:42.575034 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:42.575036 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:42.575037 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:10:42.575109 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:42.575110 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:42.575749 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:42.575751 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:42.576491 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:42.576493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:42.577043 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:42.577044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:42.577118 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:42.577121 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:10:42.577190 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 00:10:42.577232 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 00:10:42.577303 19476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:42.574830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42687 (local address 127.0.98.2:36119)
0620 00:10:42.574882 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:42.574884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:42.574893 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:10:42.574925 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:42.574925 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:42.574935 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:42.575027 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:10:42.575183 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:42.575655 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:42.576586 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:42.576941 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:42.577043 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:42.577053 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:42.577196 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 00:10:42.577232 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 00:10:42.577566 19487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:42.574763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45321 (local address 127.0.98.3:55803)
0620 00:10:42.574941 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:42.574950 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:42.575033 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:42.575231 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:42.575232 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:42.575234 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:10:42.575301 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:42.575305 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:42.575931 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:42.575932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:42.576695 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:42.576697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:42.577236 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:42.577237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:42.577306 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:42.577316 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 00:10:42.577409 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 00:10:42.577469 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":105,"threads_started":1}
I20260620 00:10:42.577654 19475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:42.574833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55803 (local address 127.0.98.1:45321)
0620 00:10:42.574879 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:42.574881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:42.575019 (+ 138us) server_negotiation.cc:407] Connection header received
0620 00:10:42.575090 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:42.575091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:42.575103 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:42.575222 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:10:42.575394 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:42.575840 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:42.576787 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:42.577153 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:42.577235 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:42.577244 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:42.577528 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 00:10:42.577573 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 00:10:42.577803 19287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94ab22af3cc448f69f823d2a99026198" candidate_uuid: "c87d0d97f45c42ce88521903c84838de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ab50bda535e4d07bf27344ef723479a" is_pre_election: true
I20260620 00:10:42.577914 19156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94ab22af3cc448f69f823d2a99026198" candidate_uuid: "c87d0d97f45c42ce88521903c84838de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f23e37fdaec4127977924bc37348346" is_pre_election: true
I20260620 00:10:42.577972 19287 raft_consensus.cc:2454] T 94ab22af3cc448f69f823d2a99026198 P 7ab50bda535e4d07bf27344ef723479a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c87d0d97f45c42ce88521903c84838de in term 0.
I20260620 00:10:42.578040 19156 raft_consensus.cc:2454] T 94ab22af3cc448f69f823d2a99026198 P 4f23e37fdaec4127977924bc37348346 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c87d0d97f45c42ce88521903c84838de in term 0.
I20260620 00:10:42.578276 19353 leader_election.cc:302] T 94ab22af3cc448f69f823d2a99026198 P c87d0d97f45c42ce88521903c84838de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ab50bda535e4d07bf27344ef723479a, c87d0d97f45c42ce88521903c84838de; no voters:
I20260620 00:10:42.578408 19486 raft_consensus.cc:2781] T 94ab22af3cc448f69f823d2a99026198 P c87d0d97f45c42ce88521903c84838de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:10:42.578462 19486 raft_consensus.cc:482] T 94ab22af3cc448f69f823d2a99026198 P c87d0d97f45c42ce88521903c84838de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:10:42.578485 19486 raft_consensus.cc:3037] T 94ab22af3cc448f69f823d2a99026198 P c87d0d97f45c42ce88521903c84838de [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:10:42.579183 19486 raft_consensus.cc:504] T 94ab22af3cc448f69f823d2a99026198 P c87d0d97f45c42ce88521903c84838de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f23e37fdaec4127977924bc37348346" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45321 } } peers { permanent_uuid: "7ab50bda535e4d07bf27344ef723479a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36119 } } peers { permanent_uuid: "c87d0d97f45c42ce88521903c84838de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35151 } }
I20260620 00:10:42.579326 19486 leader_election.cc:288] T 94ab22af3cc448f69f823d2a99026198 P c87d0d97f45c42ce88521903c84838de [CANDIDATE]: Term 1 election: Requested vote from peers 4f23e37fdaec4127977924bc37348346 (127.0.98.1:45321), 7ab50bda535e4d07bf27344ef723479a (127.0.98.2:36119)
I20260620 00:10:42.579496 19156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94ab22af3cc448f69f823d2a99026198" candidate_uuid: "c87d0d97f45c42ce88521903c84838de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f23e37fdaec4127977924bc37348346"
I20260620 00:10:42.579499 19287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94ab22af3cc448f69f823d2a99026198" candidate_uuid: "c87d0d97f45c42ce88521903c84838de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ab50bda535e4d07bf27344ef723479a"
I20260620 00:10:42.579569 19287 raft_consensus.cc:3037] T 94ab22af3cc448f69f823d2a99026198 P 7ab50bda535e4d07bf27344ef723479a [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:10:42.579569 19156 raft_consensus.cc:3037] T 94ab22af3cc448f69f823d2a99026198 P 4f23e37fdaec4127977924bc37348346 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:10:42.580449 19156 raft_consensus.cc:2454] T 94ab22af3cc448f69f823d2a99026198 P 4f23e37fdaec4127977924bc37348346 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c87d0d97f45c42ce88521903c84838de in term 1.
I20260620 00:10:42.580449 19287 raft_consensus.cc:2454] T 94ab22af3cc448f69f823d2a99026198 P 7ab50bda535e4d07bf27344ef723479a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c87d0d97f45c42ce88521903c84838de in term 1.
I20260620 00:10:42.580636 19354 leader_election.cc:302] T 94ab22af3cc448f69f823d2a99026198 P c87d0d97f45c42ce88521903c84838de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f23e37fdaec4127977924bc37348346, c87d0d97f45c42ce88521903c84838de; no voters:
I20260620 00:10:42.580750 19486 raft_consensus.cc:2781] T 94ab22af3cc448f69f823d2a99026198 P c87d0d97f45c42ce88521903c84838de [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:10:42.580928 19486 raft_consensus.cc:686] T 94ab22af3cc448f69f823d2a99026198 P c87d0d97f45c42ce88521903c84838de [term 1 LEADER]: Becoming Leader. State: Replica: c87d0d97f45c42ce88521903c84838de, State: Running, Role: LEADER
I20260620 00:10:42.581038 19486 consensus_queue.cc:237] T 94ab22af3cc448f69f823d2a99026198 P c87d0d97f45c42ce88521903c84838de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f23e37fdaec4127977924bc37348346" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45321 } } peers { permanent_uuid: "7ab50bda535e4d07bf27344ef723479a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36119 } } peers { permanent_uuid: "c87d0d97f45c42ce88521903c84838de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35151 } }
I20260620 00:10:42.582059 19013 catalog_manager.cc:5495] T 94ab22af3cc448f69f823d2a99026198 P c87d0d97f45c42ce88521903c84838de reported cstate change: term changed from 0 to 1, leader changed from <none> to c87d0d97f45c42ce88521903c84838de (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c87d0d97f45c42ce88521903c84838de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f23e37fdaec4127977924bc37348346" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ab50bda535e4d07bf27344ef723479a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c87d0d97f45c42ce88521903c84838de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35151 } health_report { overall_health: HEALTHY } } }
W20260620 00:10:42.607925 19203 tablet.cc:2369] T 94ab22af3cc448f69f823d2a99026198 P 4f23e37fdaec4127977924bc37348346: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 00:10:42.615126 19465 tablet.cc:2369] T 94ab22af3cc448f69f823d2a99026198 P c87d0d97f45c42ce88521903c84838de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:10:42.615507 19477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:42.612846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52140 (local address 127.0.98.3:35151)
0620 00:10:42.612913 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:42.612916 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:42.612935 (+ 19us) server_negotiation.cc:407] Connection header received
0620 00:10:42.612994 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:42.612995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:42.613014 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:42.613094 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:10:42.613238 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:42.613819 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:42.614648 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:42.614843 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:42.614927 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:42.615099 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:10:42.615183 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:10:42.615298 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:42.615415 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 00:10:42.615423 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 00:10:42.617247 19156 raft_consensus.cc:1261] T 94ab22af3cc448f69f823d2a99026198 P 4f23e37fdaec4127977924bc37348346 [term 1 FOLLOWER]: Refusing update from remote peer c87d0d97f45c42ce88521903c84838de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:10:42.617247 19287 raft_consensus.cc:1261] T 94ab22af3cc448f69f823d2a99026198 P 7ab50bda535e4d07bf27344ef723479a [term 1 FOLLOWER]: Refusing update from remote peer c87d0d97f45c42ce88521903c84838de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:10:42.617697 19488 consensus_queue.cc:1035] T 94ab22af3cc448f69f823d2a99026198 P c87d0d97f45c42ce88521903c84838de [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f23e37fdaec4127977924bc37348346" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:10:42.617841 19486 consensus_queue.cc:1035] T 94ab22af3cc448f69f823d2a99026198 P c87d0d97f45c42ce88521903c84838de [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ab50bda535e4d07bf27344ef723479a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:10:42.621286 19492 mvcc.cc:205] Tried to move back new op lower bound from 7298720737757515776 to 7298720737612824576. Current Snapshot: MvccSnapshot[applied={T|T < 7298720737757515776}]
I20260620 00:10:42.621459 19491 mvcc.cc:205] Tried to move back new op lower bound from 7298720737757515776 to 7298720737612824576. Current Snapshot: MvccSnapshot[applied={T|T < 7298720737757515776 or (T in {7298720737757515776})}]
I20260620 00:10:42.622864 19014 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:10:42.622956 19014 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:10:42.623710 19014 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 22d5b4e9ea0f416f993f5ef87cef709d: Sending DeleteTablet for 3 replicas of tablet 94ab22af3cc448f69f823d2a99026198
I20260620 00:10:42.624037 19136 tablet_service.cc:1514] Processing DeleteTablet for tablet 94ab22af3cc448f69f823d2a99026198 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:10:42 UTC) from {username='slave'} at 127.0.0.1:38322
I20260620 00:10:42.624070 19267 tablet_service.cc:1514] Processing DeleteTablet for tablet 94ab22af3cc448f69f823d2a99026198 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:10:42 UTC) from {username='slave'} at 127.0.0.1:35550
I20260620 00:10:42.624109 19398 tablet_service.cc:1514] Processing DeleteTablet for tablet 94ab22af3cc448f69f823d2a99026198 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:10:42 UTC) from {username='slave'} at 127.0.0.1:52132
I20260620 00:10:42.624330 19503 tablet_replica.cc:330] T 94ab22af3cc448f69f823d2a99026198 P c87d0d97f45c42ce88521903c84838de: stopping tablet replica
I20260620 00:10:42.624469 19503 raft_consensus.cc:2229] T 94ab22af3cc448f69f823d2a99026198 P c87d0d97f45c42ce88521903c84838de [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:10:42.624503 19502 tablet_replica.cc:330] T 94ab22af3cc448f69f823d2a99026198 P 7ab50bda535e4d07bf27344ef723479a: stopping tablet replica
I20260620 00:10:42.624595 19502 raft_consensus.cc:2229] T 94ab22af3cc448f69f823d2a99026198 P 7ab50bda535e4d07bf27344ef723479a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:10:42.624594 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 19074
I20260620 00:10:42.624681 19503 raft_consensus.cc:2258] T 94ab22af3cc448f69f823d2a99026198 P c87d0d97f45c42ce88521903c84838de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:10:42.626370 19502 raft_consensus.cc:2258] T 94ab22af3cc448f69f823d2a99026198 P 7ab50bda535e4d07bf27344ef723479a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:10:42.626632 19503 ts_tablet_manager.cc:1905] T 94ab22af3cc448f69f823d2a99026198 P c87d0d97f45c42ce88521903c84838de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:10:42.626821 19502 ts_tablet_manager.cc:1905] T 94ab22af3cc448f69f823d2a99026198 P 7ab50bda535e4d07bf27344ef723479a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:10:42.628381 19502 ts_tablet_manager.cc:1918] T 94ab22af3cc448f69f823d2a99026198 P 7ab50bda535e4d07bf27344ef723479a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:10:42.628425 19503 ts_tablet_manager.cc:1918] T 94ab22af3cc448f69f823d2a99026198 P c87d0d97f45c42ce88521903c84838de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:10:42.628464 19502 log.cc:1196] T 94ab22af3cc448f69f823d2a99026198 P 7ab50bda535e4d07bf27344ef723479a: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-1/wal/wals/94ab22af3cc448f69f823d2a99026198
I20260620 00:10:42.628482 19503 log.cc:1196] T 94ab22af3cc448f69f823d2a99026198 P c87d0d97f45c42ce88521903c84838de: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914001714546-392-0/minicluster-data/ts-2/wal/wals/94ab22af3cc448f69f823d2a99026198
I20260620 00:10:42.628733 19503 ts_tablet_manager.cc:1939] T 94ab22af3cc448f69f823d2a99026198 P c87d0d97f45c42ce88521903c84838de: Deleting consensus metadata
I20260620 00:10:42.628733 19502 ts_tablet_manager.cc:1939] T 94ab22af3cc448f69f823d2a99026198 P 7ab50bda535e4d07bf27344ef723479a: Deleting consensus metadata
I20260620 00:10:42.629120 19000 catalog_manager.cc:4841] TS 7ab50bda535e4d07bf27344ef723479a (127.0.98.2:36119): tablet 94ab22af3cc448f69f823d2a99026198 (table test-table [id=4414f1acbbed4341b4be09483b57df13]) successfully deleted
I20260620 00:10:42.629154 18998 catalog_manager.cc:4841] TS c87d0d97f45c42ce88521903c84838de (127.0.98.3:35151): tablet 94ab22af3cc448f69f823d2a99026198 (table test-table [id=4414f1acbbed4341b4be09483b57df13]) successfully deleted
W20260620 00:10:42.629837 19001 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45321 (error 108)
I20260620 00:10:42.630106 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 19205
I20260620 00:10:42.632893 19473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:42.632724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45321 (local address 127.0.0.1:38326)
0620 00:10:42.632799 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:42.632851 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 00:10:42.633031 19001 catalog_manager.cc:4584] TS 4f23e37fdaec4127977924bc37348346 (127.0.98.1:45321): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 94ab22af3cc448f69f823d2a99026198: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45321: connect: Connection refused (error 111)
I20260620 00:10:42.636205 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 19336
I20260620 00:10:42.640966 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 18983
2026-06-20T00:10:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (867 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:10:42 dist-test-slave-grqt krb5kdc[19507](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:10:42 dist-test-slave-grqt krb5kdc[19507](info): set up 2 sockets
Jun 20 00:10:42 dist-test-slave-grqt krb5kdc[19507](info): commencing operation
krb5kdc: starting...
W20260620 00:10:43.669721 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:10:43 dist-test-slave-grqt krb5kdc[19507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:10:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:10:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:10:43.788923 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39537
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39537
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:10:43.871301 19520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:10:43.871583 19520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:10:43.871626 19520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:10:43.872949 19520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:10:43.873044 19520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:10:43.873068 19520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:10:43.873091 19520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:10:43.873121 19520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:10:43.874822 19520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:10:43.875140 19520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:10:43.875387 19520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:10:43.880631 19525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:43.880798 19520 server_base.cc:1034] running on GCE node
W20260620 00:10:43.880640 19526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:43.880631 19528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:43.881253 19520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:10:43.881628 19520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:10:43.882824 19520 hybrid_clock.cc:648] HybridClock initialized: now 1781914243882780 us; error 60 us; skew 500 ppm
Jun 20 00:10:43 dist-test-slave-grqt krb5kdc[19507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:10:43.885689 19520 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:10:43.886193 19520 webserver.cc:460] Webserver started at http://127.0.98.62:46035/ using document root <none> and password file <none>
I20260620 00:10:43.886438 19520 fs_manager.cc:359] Metadata directory not provided
I20260620 00:10:43.886492 19520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:10:43.886669 19520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:10:43.887759 19520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "7d50441132c84957b22291999a7d435f"
format_stamp: "Formatted at 2026-06-20 00:10:43 on dist-test-slave-grqt"
server_key: "45ad6c8ca4197bcd94a209c83dd0873a"
server_key_iv: "de4040116c250c83b9b5d4e16664971b"
server_key_version: "encryptionkey@0"
I20260620 00:10:43.888125 19520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "7d50441132c84957b22291999a7d435f"
format_stamp: "Formatted at 2026-06-20 00:10:43 on dist-test-slave-grqt"
server_key: "45ad6c8ca4197bcd94a209c83dd0873a"
server_key_iv: "de4040116c250c83b9b5d4e16664971b"
server_key_version: "encryptionkey@0"
I20260620 00:10:43.889612 19520 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:10:43.890471 19535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:43.890666 19520 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:10:43.890780 19520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "7d50441132c84957b22291999a7d435f"
format_stamp: "Formatted at 2026-06-20 00:10:43 on dist-test-slave-grqt"
server_key: "45ad6c8ca4197bcd94a209c83dd0873a"
server_key_iv: "de4040116c250c83b9b5d4e16664971b"
server_key_version: "encryptionkey@0"
I20260620 00:10:43.890863 19520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:10:43.904654 19520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:10:43.906621 19520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:10:43.906792 19520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:10:43.911023 19520 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39537
I20260620 00:10:43.911062 19587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39537 every 8 connection(s)
I20260620 00:10:43.911453 19520 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:10:43.912328 19588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:10:43.914134 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 19520
I20260620 00:10:43.914273 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:10:43.914481 392 external_mini_cluster.cc:1428] Setting key 6f8746a68e3351e7be8823e217faad10
I20260620 00:10:43.914963 19588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d50441132c84957b22291999a7d435f: Bootstrap starting.
I20260620 00:10:43.915705 19588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d50441132c84957b22291999a7d435f: Neither blocks nor log segments found. Creating new log.
I20260620 00:10:43.916019 19588 log.cc:824] T 00000000000000000000000000000000 P 7d50441132c84957b22291999a7d435f: Log is configured to *not* fsync() on all Append() calls
I20260620 00:10:43.916844 19588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d50441132c84957b22291999a7d435f: No bootstrap required, opened a new log
I20260620 00:10:43.919278 19588 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d50441132c84957b22291999a7d435f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d50441132c84957b22291999a7d435f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39537 } }
I20260620 00:10:43.919461 19588 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d50441132c84957b22291999a7d435f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:10:43.919493 19588 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d50441132c84957b22291999a7d435f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d50441132c84957b22291999a7d435f, State: Initialized, Role: FOLLOWER
I20260620 00:10:43.919613 19588 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d50441132c84957b22291999a7d435f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d50441132c84957b22291999a7d435f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39537 } }
I20260620 00:10:43.919687 19588 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d50441132c84957b22291999a7d435f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:10:43.919724 19588 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d50441132c84957b22291999a7d435f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:10:43.919762 19588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d50441132c84957b22291999a7d435f [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:10:43.920514 19588 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d50441132c84957b22291999a7d435f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d50441132c84957b22291999a7d435f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39537 } }
I20260620 00:10:43.920609 19588 leader_election.cc:302] T 00000000000000000000000000000000 P 7d50441132c84957b22291999a7d435f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d50441132c84957b22291999a7d435f; no voters:
I20260620 00:10:43.920859 19588 leader_election.cc:288] T 00000000000000000000000000000000 P 7d50441132c84957b22291999a7d435f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:10:43.920884 19593 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d50441132c84957b22291999a7d435f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:10:43.921010 19593 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d50441132c84957b22291999a7d435f [term 1 LEADER]: Becoming Leader. State: Replica: 7d50441132c84957b22291999a7d435f, State: Running, Role: LEADER
I20260620 00:10:43.921139 19588 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d50441132c84957b22291999a7d435f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:10:43.921156 19593 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d50441132c84957b22291999a7d435f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d50441132c84957b22291999a7d435f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39537 } }
I20260620 00:10:43.921620 19594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d50441132c84957b22291999a7d435f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d50441132c84957b22291999a7d435f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d50441132c84957b22291999a7d435f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39537 } } }
I20260620 00:10:43.921638 19595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d50441132c84957b22291999a7d435f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d50441132c84957b22291999a7d435f. Latest consensus state: current_term: 1 leader_uuid: "7d50441132c84957b22291999a7d435f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d50441132c84957b22291999a7d435f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39537 } } }
I20260620 00:10:43.921728 19594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d50441132c84957b22291999a7d435f [sys.catalog]: This master's current role is: LEADER
I20260620 00:10:43.921741 19595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d50441132c84957b22291999a7d435f [sys.catalog]: This master's current role is: LEADER
Jun 20 00:10:43 dist-test-slave-grqt krb5kdc[19507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
W20260620 00:10:43.923017 19609 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7d50441132c84957b22291999a7d435f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:10:43.923079 19609 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:10:43.923135 19600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:10:43.923316 19600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:10:43.925071 19600 catalog_manager.cc:1347] Generated new cluster ID: 942b3ce860ec422ab97d60922edc1733
I20260620 00:10:43.925124 19600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:10:43.926761 19591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:43.915075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45846 (local address 127.0.98.62:39537)
0620 00:10:43.915259 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:43.915269 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:43.915304 (+ 35us) server_negotiation.cc:407] Connection header received
0620 00:10:43.915924 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:43.915932 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:43.916000 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:43.916168 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:43.916982 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:43.918449 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:43.919217 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:43.919834 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:43.920096 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:43.923058 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:43.923062 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:43.923062 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:10:43.923076 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:43.925181 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:43.925622 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:43.925625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:43.925625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:43.925722 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:43.925936 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:43.925937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:43.925937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:43.926187 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:43.926366 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:43.926444 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 00:10:43.926580 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
I20260620 00:10:43.957774 19600 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:10:43.958281 19600 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:10:43.964300 19600 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d50441132c84957b22291999a7d435f: Generated new TSK 0
I20260620 00:10:43.964545 19600 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:10:43.999861 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39537
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:10:44.081851 19614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:10:44.082085 19614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:10:44.082111 19614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:10:44.083323 19614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:10:44.083369 19614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:10:44.083395 19614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:10:44.084985 19614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:10:44.085249 19614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:10:44.085512 19614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 00:10:44.090603 19614 server_base.cc:1034] running on GCE node
W20260620 00:10:44.090588 19622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:44.090586 19620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:44.090586 19619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:44.091094 19614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:10:44.091354 19614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:10:44.092528 19614 hybrid_clock.cc:648] HybridClock initialized: now 1781914244092510 us; error 39 us; skew 500 ppm
Jun 20 00:10:44 dist-test-slave-grqt krb5kdc[19507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914244, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:10:44.095304 19614 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:10:44.095826 19614 webserver.cc:460] Webserver started at http://127.0.98.1:41365/ using document root <none> and password file <none>
I20260620 00:10:44.096063 19614 fs_manager.cc:359] Metadata directory not provided
I20260620 00:10:44.096132 19614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:10:44.096288 19614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:10:44.097354 19614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-0/data/instance:
uuid: "a13ea8d8dade4e0b920ace43abfb02d0"
format_stamp: "Formatted at 2026-06-20 00:10:44 on dist-test-slave-grqt"
server_key: "dfcf6b2c0d76a59341ad7a79c29fed7f"
server_key_iv: "ccdf2b9f1c539c3b6c7cb53c4ec20bcc"
server_key_version: "encryptionkey@0"
I20260620 00:10:44.097795 19614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a13ea8d8dade4e0b920ace43abfb02d0"
format_stamp: "Formatted at 2026-06-20 00:10:44 on dist-test-slave-grqt"
server_key: "dfcf6b2c0d76a59341ad7a79c29fed7f"
server_key_iv: "ccdf2b9f1c539c3b6c7cb53c4ec20bcc"
server_key_version: "encryptionkey@0"
I20260620 00:10:44.099145 19614 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:10:44.099838 19629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:44.100000 19614 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 00:10:44.100101 19614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-0/wal
uuid: "a13ea8d8dade4e0b920ace43abfb02d0"
format_stamp: "Formatted at 2026-06-20 00:10:44 on dist-test-slave-grqt"
server_key: "dfcf6b2c0d76a59341ad7a79c29fed7f"
server_key_iv: "ccdf2b9f1c539c3b6c7cb53c4ec20bcc"
server_key_version: "encryptionkey@0"
I20260620 00:10:44.100185 19614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:10:44.114789 19614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:10:44.117200 19614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:10:44.117396 19614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:10:44.117646 19614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:10:44.117962 19614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:10:44.118016 19614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:44.118069 19614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:10:44.118101 19614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:44.124330 19614 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36731
I20260620 00:10:44.124387 19742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36731 every 8 connection(s)
I20260620 00:10:44.124752 19614 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 00:10:44 dist-test-slave-grqt krb5kdc[19507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914244, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:10:44.134433 19591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:44.125573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34817 (local address 127.0.98.62:39537)
0620 00:10:44.125636 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:44.125638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:44.125871 (+ 233us) server_negotiation.cc:407] Connection header received
0620 00:10:44.126861 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:44.126863 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:44.126875 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:44.126923 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:44.127563 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:44.128086 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:44.129131 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:44.129343 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:44.129468 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:44.131550 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:44.131552 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:44.131552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:10:44.131559 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:44.132969 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:44.133446 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:44.133447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:44.133447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:44.133511 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:44.133764 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:44.133765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:44.133766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:44.133880 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:44.133955 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:44.134247 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 00:10:44.134339 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 00:10:44.134639 19745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:44.125565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39537 (local address 127.0.98.1:34817)
0620 00:10:44.125750 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:44.125787 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:44.126739 (+ 952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:44.126995 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:44.126999 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:44.127274 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:10:44.127461 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:44.127466 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:44.128185 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:44.128186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:44.129015 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:44.129017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:44.129479 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:44.129480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:44.129609 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:44.130114 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:10:44.130122 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:10:44.131454 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:10:44.133101 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:10:44.133108 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:10:44.133110 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:10:44.133333 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:10:44.133584 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:10:44.133585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:10:44.133585 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:10:44.133693 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:10:44.133956 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:10:44.133963 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:10:44.134109 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 00:10:44.134226 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 00:10:44.134378 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":78,"threads_started":1}
I20260620 00:10:44.134891 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 19614
I20260620 00:10:44.135022 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:10:44.135288 392 external_mini_cluster.cc:1428] Setting key f5e54106275c8fb96b875053e8b5c755
I20260620 00:10:44.135314 19743 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39537
I20260620 00:10:44.135429 19743 heartbeater.cc:464] Registering TS with master...
I20260620 00:10:44.135640 19743 heartbeater.cc:511] Master 127.0.98.62:39537 requested a full tablet report, sending...
I20260620 00:10:44.136211 19552 ts_manager.cc:194] Registered new tserver with Master: a13ea8d8dade4e0b920ace43abfb02d0 (127.0.98.1:36731)
I20260620 00:10:44.136828 19552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34817
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:10:44.160813 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39537
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:10:44.247798 19748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:10:44.248016 19748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:10:44.248040 19748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:10:44.249131 19748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:10:44.249167 19748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:10:44.249193 19748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:10:44.250674 19748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:10:44.250860 19748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:10:44.251040 19748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:10:44.255896 19753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:44.255900 19756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:44.255900 19754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:44.256054 19748 server_base.cc:1034] running on GCE node
I20260620 00:10:44.256335 19748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:10:44.256592 19748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:10:44.257753 19748 hybrid_clock.cc:648] HybridClock initialized: now 1781914244257738 us; error 26 us; skew 500 ppm
Jun 20 00:10:44 dist-test-slave-grqt krb5kdc[19507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914244, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:10:44.260241 19748 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:10:44.260685 19748 webserver.cc:460] Webserver started at http://127.0.98.2:43215/ using document root <none> and password file <none>
I20260620 00:10:44.260923 19748 fs_manager.cc:359] Metadata directory not provided
I20260620 00:10:44.260993 19748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:10:44.261152 19748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:10:44.262328 19748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-1/data/instance:
uuid: "a622abefc550402e9acdad0f17cf9a74"
format_stamp: "Formatted at 2026-06-20 00:10:44 on dist-test-slave-grqt"
server_key: "1acd7f1332b1987402bc546c182a3393"
server_key_iv: "c5e89c9473b1c1eaa9833643f7e33828"
server_key_version: "encryptionkey@0"
I20260620 00:10:44.262706 19748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a622abefc550402e9acdad0f17cf9a74"
format_stamp: "Formatted at 2026-06-20 00:10:44 on dist-test-slave-grqt"
server_key: "1acd7f1332b1987402bc546c182a3393"
server_key_iv: "c5e89c9473b1c1eaa9833643f7e33828"
server_key_version: "encryptionkey@0"
I20260620 00:10:44.264062 19748 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:10:44.264832 19763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:44.265053 19748 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:10:44.265158 19748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-1/wal
uuid: "a622abefc550402e9acdad0f17cf9a74"
format_stamp: "Formatted at 2026-06-20 00:10:44 on dist-test-slave-grqt"
server_key: "1acd7f1332b1987402bc546c182a3393"
server_key_iv: "c5e89c9473b1c1eaa9833643f7e33828"
server_key_version: "encryptionkey@0"
I20260620 00:10:44.265231 19748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:10:44.288179 19748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:10:44.290442 19748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:10:44.290586 19748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:10:44.290766 19748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:10:44.291110 19748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:10:44.291144 19748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:44.291193 19748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:10:44.291219 19748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:44.297219 19748 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36945
I20260620 00:10:44.297260 19876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36945 every 8 connection(s)
I20260620 00:10:44.297633 19748 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 00:10:44 dist-test-slave-grqt krb5kdc[19507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914244, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:10:44.306166 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 19748
I20260620 00:10:44.306293 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:10:44.306491 392 external_mini_cluster.cc:1428] Setting key 30e75539189bb25e28967e46320019b9
I20260620 00:10:44.307585 19591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:44.298349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56405 (local address 127.0.98.62:39537)
0620 00:10:44.298437 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:44.298440 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:44.298626 (+ 186us) server_negotiation.cc:407] Connection header received
0620 00:10:44.299547 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:44.299548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:44.299558 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:44.299611 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:44.300175 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:44.300672 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:44.301636 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:44.301820 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:44.301872 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:44.304296 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:44.304298 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:44.304298 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:10:44.304305 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:44.305861 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:44.306388 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:44.306389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:44.306389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:44.306469 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:44.306781 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:44.306783 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:44.306783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:44.306979 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:44.307049 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:44.307455 (+ 406us) server_negotiation.cc:299] Negotiation successful
0620 00:10:44.307521 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 00:10:44.307932 19879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:44.298313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39537 (local address 127.0.98.2:56405)
0620 00:10:44.298506 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:44.298542 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:44.299439 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:44.299664 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:44.299668 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:44.299914 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:10:44.300106 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:44.300110 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:44.300746 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:44.300747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:44.301567 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:44.301569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:44.301888 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:44.301889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:44.301999 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:44.302548 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:10:44.302556 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:10:44.304118 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:10:44.305959 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:10:44.305968 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:10:44.305971 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:10:44.306304 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:10:44.306546 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:10:44.306547 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:10:44.306547 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:10:44.306711 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:10:44.307070 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:10:44.307076 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:10:44.307302 (+ 226us) client_negotiation.cc:769] Sending connection context
0620 00:10:44.307431 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 00:10:44.307645 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":114,"threads_started":1}
I20260620 00:10:44.308487 19877 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39537
I20260620 00:10:44.308581 19877 heartbeater.cc:464] Registering TS with master...
I20260620 00:10:44.308786 19877 heartbeater.cc:511] Master 127.0.98.62:39537 requested a full tablet report, sending...
I20260620 00:10:44.309224 19552 ts_manager.cc:194] Registered new tserver with Master: a622abefc550402e9acdad0f17cf9a74 (127.0.98.2:36945)
I20260620 00:10:44.309644 19552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56405
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:10:44.332646 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39537
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:10:44.414546 19882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:10:44.414799 19882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:10:44.414841 19882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:10:44.416020 19882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:10:44.416090 19882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:10:44.416136 19882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:10:44.417683 19882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:10:44.417948 19882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:10:44.418179 19882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:10:44.423030 19887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:44.423030 19888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:44.423051 19890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:44.423498 19882 server_base.cc:1034] running on GCE node
I20260620 00:10:44.423725 19882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:10:44.423964 19882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:10:44.425143 19882 hybrid_clock.cc:648] HybridClock initialized: now 1781914244425130 us; error 43 us; skew 500 ppm
Jun 20 00:10:44 dist-test-slave-grqt krb5kdc[19507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914244, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:10:44.427663 19882 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:10:44.428160 19882 webserver.cc:460] Webserver started at http://127.0.98.3:39419/ using document root <none> and password file <none>
I20260620 00:10:44.428385 19882 fs_manager.cc:359] Metadata directory not provided
I20260620 00:10:44.428452 19882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:10:44.428609 19882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:10:44.429728 19882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-2/data/instance:
uuid: "143ad93e84114fb5bcfb9ad87b3d2ffa"
format_stamp: "Formatted at 2026-06-20 00:10:44 on dist-test-slave-grqt"
server_key: "76633b2ccfc1db2135c0fe88447373f4"
server_key_iv: "bb947bee023af214de357316806fe8d4"
server_key_version: "encryptionkey@0"
I20260620 00:10:44.430085 19882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-2/wal/instance:
uuid: "143ad93e84114fb5bcfb9ad87b3d2ffa"
format_stamp: "Formatted at 2026-06-20 00:10:44 on dist-test-slave-grqt"
server_key: "76633b2ccfc1db2135c0fe88447373f4"
server_key_iv: "bb947bee023af214de357316806fe8d4"
server_key_version: "encryptionkey@0"
I20260620 00:10:44.431416 19882 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:10:44.432111 19897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:44.432282 19882 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:10:44.432379 19882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-2/wal
uuid: "143ad93e84114fb5bcfb9ad87b3d2ffa"
format_stamp: "Formatted at 2026-06-20 00:10:44 on dist-test-slave-grqt"
server_key: "76633b2ccfc1db2135c0fe88447373f4"
server_key_iv: "bb947bee023af214de357316806fe8d4"
server_key_version: "encryptionkey@0"
I20260620 00:10:44.432461 19882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:10:44.451138 19882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:10:44.453434 19882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:10:44.453624 19882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:10:44.453863 19882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:10:44.454212 19882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:10:44.454247 19882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:44.454290 19882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:10:44.454317 19882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:44.460004 19882 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38969
I20260620 00:10:44.460076 20010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38969 every 8 connection(s)
I20260620 00:10:44.460408 19882 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 00:10:44 dist-test-slave-grqt krb5kdc[19507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914244, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:10:44.467491 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 19882
I20260620 00:10:44.467640 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:10:44.467826 392 external_mini_cluster.cc:1428] Setting key 5c491106e5ebf10b1fead4a26e5959de
I20260620 00:10:44.470713 19591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:44.461211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52589 (local address 127.0.98.62:39537)
0620 00:10:44.461255 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:44.461257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:44.461518 (+ 261us) server_negotiation.cc:407] Connection header received
0620 00:10:44.462500 (+ 982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:44.462502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:44.462514 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:44.462572 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:44.463241 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:44.463756 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:44.464922 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:44.465141 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:44.465228 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:44.467630 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:44.467632 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:44.467632 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:10:44.467644 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:44.469168 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:44.469748 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:44.469749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:44.469749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:44.469813 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:44.470105 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:44.470106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:44.470106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:44.470232 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:44.470293 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:44.470572 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 00:10:44.470647 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 00:10:44.470863 20013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:44.461171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39537 (local address 127.0.98.3:52589)
0620 00:10:44.461342 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:44.461426 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:44.462382 (+ 956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:44.462657 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:44.462661 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:44.462932 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:10:44.463131 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:44.463135 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:44.463855 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:44.463856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:44.464818 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:44.464821 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:44.465240 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:44.465241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:44.465409 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:44.465965 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:10:44.465974 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:10:44.467488 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:10:44.469271 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:10:44.469279 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:10:44.469281 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:10:44.469654 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:10:44.469890 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:10:44.469891 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:10:44.469891 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:10:44.470001 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:10:44.470297 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:10:44.470301 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:10:44.470417 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 00:10:44.470499 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 00:10:44.470641 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
I20260620 00:10:44.471329 20011 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39537
I20260620 00:10:44.471408 20011 heartbeater.cc:464] Registering TS with master...
I20260620 00:10:44.471546 20011 heartbeater.cc:511] Master 127.0.98.62:39537 requested a full tablet report, sending...
I20260620 00:10:44.471850 19552 ts_manager.cc:194] Registered new tserver with Master: 143ad93e84114fb5bcfb9ad87b3d2ffa (127.0.98.3:38969)
I20260620 00:10:44.472254 19552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52589
I20260620 00:10:44.479507 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:10:44.487543 19591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:44.480961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45848 (local address 127.0.98.62:39537)
0620 00:10:44.481066 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:44.481068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:44.481169 (+ 101us) server_negotiation.cc:407] Connection header received
0620 00:10:44.481256 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:44.481257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:44.481274 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:44.481339 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:44.482028 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:44.482530 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:44.483292 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:44.483488 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:44.483575 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:44.484867 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:44.484868 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:44.484868 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:10:44.484876 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:44.486338 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:44.486708 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:44.486710 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:44.486711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:44.486765 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:44.486972 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:44.486973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:44.486973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:44.487132 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:44.487241 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:44.487352 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 00:10:44.487437 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 00:10:44.489215 19552 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:10:44.490855 19552 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:10:44.498068 20024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:44.493225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36731 (local address 127.0.0.1:54702)
0620 00:10:44.494070 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:44.494081 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:44.494142 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:44.494356 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:44.494357 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:44.494359 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:10:44.494413 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:44.494415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:44.495711 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:44.495712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:44.496780 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:44.496782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:44.497744 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:44.497745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:44.497819 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:44.497833 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 00:10:44.497906 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 00:10:44.497956 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"thread_start_us":92,"threads_started":1}
I20260620 00:10:44.498066 20026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:44.493702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54702 (local address 127.0.98.1:36731)
0620 00:10:44.493859 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:44.493862 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:44.494136 (+ 274us) server_negotiation.cc:407] Connection header received
0620 00:10:44.494212 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:44.494214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:44.494231 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:44.494356 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:10:44.494479 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:44.495620 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:44.496878 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:44.497634 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:44.497743 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:44.497794 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:44.497932 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 00:10:44.497981 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":60,"threads_started":1}
I20260620 00:10:44.498364 20025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:44.493484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48904 (local address 127.0.98.3:38969)
0620 00:10:44.493707 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:44.493711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:44.494325 (+ 614us) server_negotiation.cc:407] Connection header received
0620 00:10:44.494365 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:44.494367 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:44.494420 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:44.494490 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:10:44.494714 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:44.495379 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:44.496878 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:44.497654 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:44.497717 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:44.497779 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:44.498109 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 00:10:44.498268 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":104,"threads_started":1}
I20260620 00:10:44.498517 20022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:44.493225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38969 (local address 127.0.0.1:48904)
0620 00:10:44.494221 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:44.494230 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:44.494273 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:44.494602 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:44.494603 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:44.494604 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:10:44.494644 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:44.494646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:44.495502 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:44.495503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:44.496780 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:44.496782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:44.497814 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:44.497815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:44.497906 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:44.497912 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 00:10:44.497965 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 00:10:44.498005 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":986,"mutex_wait_us":2,"thread_start_us":85,"threads_started":1}
I20260620 00:10:44.498773 20023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:44.493215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36945 (local address 127.0.0.1:36062)
0620 00:10:44.493387 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:44.493426 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:44.493514 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:44.494245 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:44.494249 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:44.494252 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:10:44.494337 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:44.494340 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:44.496881 (+ 2541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:44.496882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:44.497599 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:44.497601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:44.498564 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:44.498565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:44.498633 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:44.498637 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:10:44.498667 (+ 30us) client_negotiation.cc:240] Negotiation successful
0620 00:10:44.498702 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":55,"threads_started":1}
I20260620 00:10:44.499163 20027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:44.493882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36062 (local address 127.0.98.2:36945)
0620 00:10:44.494048 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:44.494052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:44.494064 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:10:44.494107 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:44.494110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:44.494126 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:44.494235 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:10:44.494459 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:44.495599 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:44.497683 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:44.498452 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:44.498977 (+ 525us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:44.499044 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:44.499070 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 00:10:44.499108 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":96,"threads_started":1}
I20260620 00:10:44.499181 19677 tablet_service.cc:1467] Processing CreateTablet for tablet e544e43af2c948098c77c093ff2884a2 (DEFAULT_TABLE table=test-table [id=afbbb702ed9d4434b0dde472a48ed6fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:10:44.499174 19945 tablet_service.cc:1467] Processing CreateTablet for tablet e544e43af2c948098c77c093ff2884a2 (DEFAULT_TABLE table=test-table [id=afbbb702ed9d4434b0dde472a48ed6fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:10:44.499527 19945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e544e43af2c948098c77c093ff2884a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:10:44.499527 19677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e544e43af2c948098c77c093ff2884a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:10:44.499686 19811 tablet_service.cc:1467] Processing CreateTablet for tablet e544e43af2c948098c77c093ff2884a2 (DEFAULT_TABLE table=test-table [id=afbbb702ed9d4434b0dde472a48ed6fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:10:44.499925 19811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e544e43af2c948098c77c093ff2884a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:10:44.502565 20028 tablet_bootstrap.cc:492] T e544e43af2c948098c77c093ff2884a2 P a622abefc550402e9acdad0f17cf9a74: Bootstrap starting.
I20260620 00:10:44.503142 20029 tablet_bootstrap.cc:492] T e544e43af2c948098c77c093ff2884a2 P 143ad93e84114fb5bcfb9ad87b3d2ffa: Bootstrap starting.
I20260620 00:10:44.503331 20028 tablet_bootstrap.cc:654] T e544e43af2c948098c77c093ff2884a2 P a622abefc550402e9acdad0f17cf9a74: Neither blocks nor log segments found. Creating new log.
I20260620 00:10:44.503870 20029 tablet_bootstrap.cc:654] T e544e43af2c948098c77c093ff2884a2 P 143ad93e84114fb5bcfb9ad87b3d2ffa: Neither blocks nor log segments found. Creating new log.
I20260620 00:10:44.503978 20028 log.cc:824] T e544e43af2c948098c77c093ff2884a2 P a622abefc550402e9acdad0f17cf9a74: Log is configured to *not* fsync() on all Append() calls
I20260620 00:10:44.504514 20029 log.cc:824] T e544e43af2c948098c77c093ff2884a2 P 143ad93e84114fb5bcfb9ad87b3d2ffa: Log is configured to *not* fsync() on all Append() calls
I20260620 00:10:44.504812 20028 tablet_bootstrap.cc:492] T e544e43af2c948098c77c093ff2884a2 P a622abefc550402e9acdad0f17cf9a74: No bootstrap required, opened a new log
I20260620 00:10:44.504859 20030 tablet_bootstrap.cc:492] T e544e43af2c948098c77c093ff2884a2 P a13ea8d8dade4e0b920ace43abfb02d0: Bootstrap starting.
I20260620 00:10:44.504886 20028 ts_tablet_manager.cc:1397] T e544e43af2c948098c77c093ff2884a2 P a622abefc550402e9acdad0f17cf9a74: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:10:44.505261 20029 tablet_bootstrap.cc:492] T e544e43af2c948098c77c093ff2884a2 P 143ad93e84114fb5bcfb9ad87b3d2ffa: No bootstrap required, opened a new log
I20260620 00:10:44.505329 20029 ts_tablet_manager.cc:1397] T e544e43af2c948098c77c093ff2884a2 P 143ad93e84114fb5bcfb9ad87b3d2ffa: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:10:44.506946 20028 raft_consensus.cc:348] T e544e43af2c948098c77c093ff2884a2 P a622abefc550402e9acdad0f17cf9a74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "143ad93e84114fb5bcfb9ad87b3d2ffa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38969 } } peers { permanent_uuid: "a13ea8d8dade4e0b920ace43abfb02d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36731 } } peers { permanent_uuid: "a622abefc550402e9acdad0f17cf9a74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36945 } }
I20260620 00:10:44.507150 20028 raft_consensus.cc:374] T e544e43af2c948098c77c093ff2884a2 P a622abefc550402e9acdad0f17cf9a74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:10:44.507189 20028 raft_consensus.cc:729] T e544e43af2c948098c77c093ff2884a2 P a622abefc550402e9acdad0f17cf9a74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a622abefc550402e9acdad0f17cf9a74, State: Initialized, Role: FOLLOWER
I20260620 00:10:44.507318 20028 consensus_queue.cc:260] T e544e43af2c948098c77c093ff2884a2 P a622abefc550402e9acdad0f17cf9a74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "143ad93e84114fb5bcfb9ad87b3d2ffa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38969 } } peers { permanent_uuid: "a13ea8d8dade4e0b920ace43abfb02d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36731 } } peers { permanent_uuid: "a622abefc550402e9acdad0f17cf9a74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36945 } }
I20260620 00:10:44.507280 20029 raft_consensus.cc:348] T e544e43af2c948098c77c093ff2884a2 P 143ad93e84114fb5bcfb9ad87b3d2ffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "143ad93e84114fb5bcfb9ad87b3d2ffa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38969 } } peers { permanent_uuid: "a13ea8d8dade4e0b920ace43abfb02d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36731 } } peers { permanent_uuid: "a622abefc550402e9acdad0f17cf9a74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36945 } }
I20260620 00:10:44.507423 20029 raft_consensus.cc:374] T e544e43af2c948098c77c093ff2884a2 P 143ad93e84114fb5bcfb9ad87b3d2ffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:10:44.507480 20029 raft_consensus.cc:729] T e544e43af2c948098c77c093ff2884a2 P 143ad93e84114fb5bcfb9ad87b3d2ffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 143ad93e84114fb5bcfb9ad87b3d2ffa, State: Initialized, Role: FOLLOWER
I20260620 00:10:44.507604 20028 ts_tablet_manager.cc:1428] T e544e43af2c948098c77c093ff2884a2 P a622abefc550402e9acdad0f17cf9a74: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:10:44.507599 20029 consensus_queue.cc:260] T e544e43af2c948098c77c093ff2884a2 P 143ad93e84114fb5bcfb9ad87b3d2ffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "143ad93e84114fb5bcfb9ad87b3d2ffa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38969 } } peers { permanent_uuid: "a13ea8d8dade4e0b920ace43abfb02d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36731 } } peers { permanent_uuid: "a622abefc550402e9acdad0f17cf9a74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36945 } }
I20260620 00:10:44.507728 19877 heartbeater.cc:503] Master 127.0.98.62:39537 was elected leader, sending a full tablet report...
I20260620 00:10:44.507884 20029 ts_tablet_manager.cc:1428] T e544e43af2c948098c77c093ff2884a2 P 143ad93e84114fb5bcfb9ad87b3d2ffa: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 00:10:44.507918 20011 heartbeater.cc:503] Master 127.0.98.62:39537 was elected leader, sending a full tablet report...
I20260620 00:10:44.508615 20030 tablet_bootstrap.cc:654] T e544e43af2c948098c77c093ff2884a2 P a13ea8d8dade4e0b920ace43abfb02d0: Neither blocks nor log segments found. Creating new log.
I20260620 00:10:44.509212 20030 log.cc:824] T e544e43af2c948098c77c093ff2884a2 P a13ea8d8dade4e0b920ace43abfb02d0: Log is configured to *not* fsync() on all Append() calls
I20260620 00:10:44.510001 20030 tablet_bootstrap.cc:492] T e544e43af2c948098c77c093ff2884a2 P a13ea8d8dade4e0b920ace43abfb02d0: No bootstrap required, opened a new log
I20260620 00:10:44.510110 20030 ts_tablet_manager.cc:1397] T e544e43af2c948098c77c093ff2884a2 P a13ea8d8dade4e0b920ace43abfb02d0: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 00:10:44.511639 20030 raft_consensus.cc:348] T e544e43af2c948098c77c093ff2884a2 P a13ea8d8dade4e0b920ace43abfb02d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "143ad93e84114fb5bcfb9ad87b3d2ffa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38969 } } peers { permanent_uuid: "a13ea8d8dade4e0b920ace43abfb02d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36731 } } peers { permanent_uuid: "a622abefc550402e9acdad0f17cf9a74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36945 } }
I20260620 00:10:44.511818 20030 raft_consensus.cc:374] T e544e43af2c948098c77c093ff2884a2 P a13ea8d8dade4e0b920ace43abfb02d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:10:44.511868 20030 raft_consensus.cc:729] T e544e43af2c948098c77c093ff2884a2 P a13ea8d8dade4e0b920ace43abfb02d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a13ea8d8dade4e0b920ace43abfb02d0, State: Initialized, Role: FOLLOWER
I20260620 00:10:44.512044 20030 consensus_queue.cc:260] T e544e43af2c948098c77c093ff2884a2 P a13ea8d8dade4e0b920ace43abfb02d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "143ad93e84114fb5bcfb9ad87b3d2ffa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38969 } } peers { permanent_uuid: "a13ea8d8dade4e0b920ace43abfb02d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36731 } } peers { permanent_uuid: "a622abefc550402e9acdad0f17cf9a74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36945 } }
I20260620 00:10:44.512359 19743 heartbeater.cc:503] Master 127.0.98.62:39537 was elected leader, sending a full tablet report...
I20260620 00:10:44.512372 20030 ts_tablet_manager.cc:1428] T e544e43af2c948098c77c093ff2884a2 P a13ea8d8dade4e0b920ace43abfb02d0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20260620 00:10:44.548162 19878 tablet.cc:2369] T e544e43af2c948098c77c093ff2884a2 P a622abefc550402e9acdad0f17cf9a74: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 00:10:44.625561 19744 tablet.cc:2369] T e544e43af2c948098c77c093ff2884a2 P a13ea8d8dade4e0b920ace43abfb02d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 00:10:44.711014 20012 tablet.cc:2369] T e544e43af2c948098c77c093ff2884a2 P 143ad93e84114fb5bcfb9ad87b3d2ffa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:10:44.733258 20033 raft_consensus.cc:482] T e544e43af2c948098c77c093ff2884a2 P a622abefc550402e9acdad0f17cf9a74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:10:44.733412 20033 raft_consensus.cc:504] T e544e43af2c948098c77c093ff2884a2 P a622abefc550402e9acdad0f17cf9a74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "143ad93e84114fb5bcfb9ad87b3d2ffa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38969 } } peers { permanent_uuid: "a13ea8d8dade4e0b920ace43abfb02d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36731 } } peers { permanent_uuid: "a622abefc550402e9acdad0f17cf9a74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36945 } }
I20260620 00:10:44.733769 20033 leader_election.cc:288] T e544e43af2c948098c77c093ff2884a2 P a622abefc550402e9acdad0f17cf9a74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 143ad93e84114fb5bcfb9ad87b3d2ffa (127.0.98.3:38969), a13ea8d8dade4e0b920ace43abfb02d0 (127.0.98.1:36731)
I20260620 00:10:44.737994 20025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:44.734065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44273 (local address 127.0.98.3:38969)
0620 00:10:44.734118 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:44.734126 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:44.734259 (+ 133us) server_negotiation.cc:407] Connection header received
0620 00:10:44.734314 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:44.734315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:44.734324 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:44.734377 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:10:44.734529 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:44.735308 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:44.736913 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:44.737521 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:44.737609 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:44.737620 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:44.737857 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 00:10:44.737899 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 00:10:44.737972 20026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:44.734064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41987 (local address 127.0.98.1:36731)
0620 00:10:44.734132 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:44.734134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:44.734144 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:10:44.734182 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:44.734183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:44.734193 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:44.734284 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:10:44.734482 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:44.735267 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:44.736913 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:44.737526 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:44.737609 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:44.737620 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:44.737857 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 00:10:44.737899 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 00:10:44.738003 19879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:44.733986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36731 (local address 127.0.98.2:41987)
0620 00:10:44.734047 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:44.734060 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:44.734106 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:44.734303 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:44.734304 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:44.734307 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:10:44.734402 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:44.734403 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:44.735413 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:44.735415 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:44.736786 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:44.736790 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:44.737659 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:44.737660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:44.737774 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:44.737778 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:10:44.737831 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 00:10:44.737889 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":52}
I20260620 00:10:44.738009 20037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:44.734010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38969 (local address 127.0.98.2:44273)
0620 00:10:44.734182 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:44.734190 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:44.734243 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:44.734398 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:44.734399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:44.734400 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:10:44.734459 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:44.734460 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:44.735413 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:44.735415 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:44.736786 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:44.736790 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:44.737659 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:44.737660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:44.737749 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:44.737755 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 00:10:44.737831 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 00:10:44.737889 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":91,"thread_start_us":114,"threads_started":1}
I20260620 00:10:44.738440 19697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e544e43af2c948098c77c093ff2884a2" candidate_uuid: "a622abefc550402e9acdad0f17cf9a74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a13ea8d8dade4e0b920ace43abfb02d0" is_pre_election: true
I20260620 00:10:44.738442 19965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e544e43af2c948098c77c093ff2884a2" candidate_uuid: "a622abefc550402e9acdad0f17cf9a74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "143ad93e84114fb5bcfb9ad87b3d2ffa" is_pre_election: true
I20260620 00:10:44.738619 19697 raft_consensus.cc:2454] T e544e43af2c948098c77c093ff2884a2 P a13ea8d8dade4e0b920ace43abfb02d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a622abefc550402e9acdad0f17cf9a74 in term 0.
I20260620 00:10:44.738631 19965 raft_consensus.cc:2454] T e544e43af2c948098c77c093ff2884a2 P 143ad93e84114fb5bcfb9ad87b3d2ffa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a622abefc550402e9acdad0f17cf9a74 in term 0.
I20260620 00:10:44.738905 19765 leader_election.cc:302] T e544e43af2c948098c77c093ff2884a2 P a622abefc550402e9acdad0f17cf9a74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 143ad93e84114fb5bcfb9ad87b3d2ffa, a622abefc550402e9acdad0f17cf9a74; no voters:
I20260620 00:10:44.739059 20033 raft_consensus.cc:2781] T e544e43af2c948098c77c093ff2884a2 P a622abefc550402e9acdad0f17cf9a74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:10:44.739111 20033 raft_consensus.cc:482] T e544e43af2c948098c77c093ff2884a2 P a622abefc550402e9acdad0f17cf9a74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:10:44.739137 20033 raft_consensus.cc:3037] T e544e43af2c948098c77c093ff2884a2 P a622abefc550402e9acdad0f17cf9a74 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:10:44.739807 20033 raft_consensus.cc:504] T e544e43af2c948098c77c093ff2884a2 P a622abefc550402e9acdad0f17cf9a74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "143ad93e84114fb5bcfb9ad87b3d2ffa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38969 } } peers { permanent_uuid: "a13ea8d8dade4e0b920ace43abfb02d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36731 } } peers { permanent_uuid: "a622abefc550402e9acdad0f17cf9a74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36945 } }
I20260620 00:10:44.739959 20033 leader_election.cc:288] T e544e43af2c948098c77c093ff2884a2 P a622abefc550402e9acdad0f17cf9a74 [CANDIDATE]: Term 1 election: Requested vote from peers 143ad93e84114fb5bcfb9ad87b3d2ffa (127.0.98.3:38969), a13ea8d8dade4e0b920ace43abfb02d0 (127.0.98.1:36731)
I20260620 00:10:44.740159 19697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e544e43af2c948098c77c093ff2884a2" candidate_uuid: "a622abefc550402e9acdad0f17cf9a74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a13ea8d8dade4e0b920ace43abfb02d0"
I20260620 00:10:44.740159 19965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e544e43af2c948098c77c093ff2884a2" candidate_uuid: "a622abefc550402e9acdad0f17cf9a74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "143ad93e84114fb5bcfb9ad87b3d2ffa"
I20260620 00:10:44.740252 19965 raft_consensus.cc:3037] T e544e43af2c948098c77c093ff2884a2 P 143ad93e84114fb5bcfb9ad87b3d2ffa [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:10:44.740252 19697 raft_consensus.cc:3037] T e544e43af2c948098c77c093ff2884a2 P a13ea8d8dade4e0b920ace43abfb02d0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:10:44.741142 19697 raft_consensus.cc:2454] T e544e43af2c948098c77c093ff2884a2 P a13ea8d8dade4e0b920ace43abfb02d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a622abefc550402e9acdad0f17cf9a74 in term 1.
I20260620 00:10:44.741142 19965 raft_consensus.cc:2454] T e544e43af2c948098c77c093ff2884a2 P 143ad93e84114fb5bcfb9ad87b3d2ffa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a622abefc550402e9acdad0f17cf9a74 in term 1.
I20260620 00:10:44.741344 19767 leader_election.cc:302] T e544e43af2c948098c77c093ff2884a2 P a622abefc550402e9acdad0f17cf9a74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a13ea8d8dade4e0b920ace43abfb02d0, a622abefc550402e9acdad0f17cf9a74; no voters:
I20260620 00:10:44.741474 20033 raft_consensus.cc:2781] T e544e43af2c948098c77c093ff2884a2 P a622abefc550402e9acdad0f17cf9a74 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:10:44.741636 20033 raft_consensus.cc:686] T e544e43af2c948098c77c093ff2884a2 P a622abefc550402e9acdad0f17cf9a74 [term 1 LEADER]: Becoming Leader. State: Replica: a622abefc550402e9acdad0f17cf9a74, State: Running, Role: LEADER
I20260620 00:10:44.741750 20033 consensus_queue.cc:237] T e544e43af2c948098c77c093ff2884a2 P a622abefc550402e9acdad0f17cf9a74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "143ad93e84114fb5bcfb9ad87b3d2ffa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38969 } } peers { permanent_uuid: "a13ea8d8dade4e0b920ace43abfb02d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36731 } } peers { permanent_uuid: "a622abefc550402e9acdad0f17cf9a74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36945 } }
I20260620 00:10:44.742689 19552 catalog_manager.cc:5495] T e544e43af2c948098c77c093ff2884a2 P a622abefc550402e9acdad0f17cf9a74 reported cstate change: term changed from 0 to 1, leader changed from <none> to a622abefc550402e9acdad0f17cf9a74 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a622abefc550402e9acdad0f17cf9a74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "143ad93e84114fb5bcfb9ad87b3d2ffa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a13ea8d8dade4e0b920ace43abfb02d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a622abefc550402e9acdad0f17cf9a74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36945 } health_report { overall_health: HEALTHY } } }
I20260620 00:10:44.780119 20027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:44.777593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36070 (local address 127.0.98.2:36945)
0620 00:10:44.777655 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:44.777657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:44.777666 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:10:44.777708 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:44.777709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:44.777732 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:44.777820 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:10:44.777957 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:44.778437 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:44.779266 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:44.779487 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:44.779575 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:44.779749 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:10:44.779863 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:10:44.779960 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:44.779990 (+ 30us) server_negotiation.cc:299] Negotiation successful
0620 00:10:44.780011 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 00:10:44.782254 19697 raft_consensus.cc:1261] T e544e43af2c948098c77c093ff2884a2 P a13ea8d8dade4e0b920ace43abfb02d0 [term 1 FOLLOWER]: Refusing update from remote peer a622abefc550402e9acdad0f17cf9a74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:10:44.782254 19965 raft_consensus.cc:1261] T e544e43af2c948098c77c093ff2884a2 P 143ad93e84114fb5bcfb9ad87b3d2ffa [term 1 FOLLOWER]: Refusing update from remote peer a622abefc550402e9acdad0f17cf9a74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:10:44.782624 20038 consensus_queue.cc:1035] T e544e43af2c948098c77c093ff2884a2 P a622abefc550402e9acdad0f17cf9a74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a13ea8d8dade4e0b920ace43abfb02d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:10:44.782793 20033 consensus_queue.cc:1035] T e544e43af2c948098c77c093ff2884a2 P a622abefc550402e9acdad0f17cf9a74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "143ad93e84114fb5bcfb9ad87b3d2ffa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:10:44.785440 20043 mvcc.cc:205] Tried to move back new op lower bound from 7298720746624286720 to 7298720746462982144. Current Snapshot: MvccSnapshot[applied={T|T < 7298720746624286720}]
I20260620 00:10:44.787988 19552 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:10:44.788085 19552 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:10:44.788823 19552 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7d50441132c84957b22291999a7d435f: Sending DeleteTablet for 3 replicas of tablet e544e43af2c948098c77c093ff2884a2
I20260620 00:10:44.789160 19811 tablet_service.cc:1514] Processing DeleteTablet for tablet e544e43af2c948098c77c093ff2884a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:10:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36062
I20260620 00:10:44.789171 19677 tablet_service.cc:1514] Processing DeleteTablet for tablet e544e43af2c948098c77c093ff2884a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:10:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54702
I20260620 00:10:44.789170 19945 tablet_service.cc:1514] Processing DeleteTablet for tablet e544e43af2c948098c77c093ff2884a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:10:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48904
I20260620 00:10:44.789434 20051 tablet_replica.cc:330] T e544e43af2c948098c77c093ff2884a2 P 143ad93e84114fb5bcfb9ad87b3d2ffa: stopping tablet replica
I20260620 00:10:44.789547 20051 raft_consensus.cc:2229] T e544e43af2c948098c77c093ff2884a2 P 143ad93e84114fb5bcfb9ad87b3d2ffa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:10:44.789644 20051 raft_consensus.cc:2258] T e544e43af2c948098c77c093ff2884a2 P 143ad93e84114fb5bcfb9ad87b3d2ffa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:10:44.789746 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 19614
I20260620 00:10:44.791383 20053 tablet_replica.cc:330] T e544e43af2c948098c77c093ff2884a2 P a622abefc550402e9acdad0f17cf9a74: stopping tablet replica
I20260620 00:10:44.791496 20053 raft_consensus.cc:2229] T e544e43af2c948098c77c093ff2884a2 P a622abefc550402e9acdad0f17cf9a74 [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:10:44.791605 20053 raft_consensus.cc:2258] T e544e43af2c948098c77c093ff2884a2 P a622abefc550402e9acdad0f17cf9a74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:10:44.792626 20051 ts_tablet_manager.cc:1905] T e544e43af2c948098c77c093ff2884a2 P 143ad93e84114fb5bcfb9ad87b3d2ffa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:10:44.793783 20053 ts_tablet_manager.cc:1905] T e544e43af2c948098c77c093ff2884a2 P a622abefc550402e9acdad0f17cf9a74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:10:44.794811 20051 ts_tablet_manager.cc:1918] T e544e43af2c948098c77c093ff2884a2 P 143ad93e84114fb5bcfb9ad87b3d2ffa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:10:44.794889 20051 log.cc:1196] T e544e43af2c948098c77c093ff2884a2 P 143ad93e84114fb5bcfb9ad87b3d2ffa: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-2/wal/wals/e544e43af2c948098c77c093ff2884a2
I20260620 00:10:44.795168 20051 ts_tablet_manager.cc:1939] T e544e43af2c948098c77c093ff2884a2 P 143ad93e84114fb5bcfb9ad87b3d2ffa: Deleting consensus metadata
I20260620 00:10:44.795598 19537 catalog_manager.cc:4841] TS 143ad93e84114fb5bcfb9ad87b3d2ffa (127.0.98.3:38969): tablet e544e43af2c948098c77c093ff2884a2 (table test-table [id=afbbb702ed9d4434b0dde472a48ed6fa]) successfully deleted
I20260620 00:10:44.795953 20053 ts_tablet_manager.cc:1918] T e544e43af2c948098c77c093ff2884a2 P a622abefc550402e9acdad0f17cf9a74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:10:44.796030 20053 log.cc:1196] T e544e43af2c948098c77c093ff2884a2 P a622abefc550402e9acdad0f17cf9a74: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914001714546-392-0/minicluster-data/ts-1/wal/wals/e544e43af2c948098c77c093ff2884a2
I20260620 00:10:44.796274 20053 ts_tablet_manager.cc:1939] T e544e43af2c948098c77c093ff2884a2 P a622abefc550402e9acdad0f17cf9a74: Deleting consensus metadata
W20260620 00:10:44.796334 19539 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36731 (error 108)
I20260620 00:10:44.796701 19538 catalog_manager.cc:4841] TS a622abefc550402e9acdad0f17cf9a74 (127.0.98.2:36945): tablet e544e43af2c948098c77c093ff2884a2 (table test-table [id=afbbb702ed9d4434b0dde472a48ed6fa]) successfully deleted
I20260620 00:10:44.796859 20022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:44.796718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36731 (local address 127.0.0.1:54706)
0620 00:10:44.796768 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:44.796821 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 00:10:44.796985 19539 catalog_manager.cc:4584] TS a13ea8d8dade4e0b920ace43abfb02d0 (127.0.98.1:36731): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e544e43af2c948098c77c093ff2884a2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36731: connect: Connection refused (error 111)
I20260620 00:10:44.797253 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 19748
I20260620 00:10:44.802039 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 19882
I20260620 00:10:44.806720 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 19520
2026-06-20T00:10:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2166 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T00:10:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:10:44Z Disabled control of system clock
I20260620 00:10:44.834002 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43079
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 00:10:44.913661 20058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:10:44.913878 20058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:10:44.913899 20058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:10:44.914981 20058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:10:44.915015 20058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:10:44.915027 20058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:10:44.915040 20058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:10:44.915052 20058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:10:44.916306 20058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43079
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:10:44.916494 20058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:10:44.916724 20058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:10:44.921818 20066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:44.921873 20063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:44.921818 20064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:44.921998 20058 server_base.cc:1034] running on GCE node
I20260620 00:10:44.922289 20058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:10:44.922653 20058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:10:44.923851 20058 hybrid_clock.cc:648] HybridClock initialized: now 1781914244923838 us; error 47 us; skew 500 ppm
I20260620 00:10:44.925184 20058 webserver.cc:460] Webserver started at http://127.0.98.62:42757/ using document root <none> and password file <none>
I20260620 00:10:44.925504 20058 fs_manager.cc:359] Metadata directory not provided
I20260620 00:10:44.925577 20058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:10:44.925737 20058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:10:44.926810 20058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "bb06fda903414ed5afcc41c49a40908e"
format_stamp: "Formatted at 2026-06-20 00:10:44 on dist-test-slave-grqt"
server_key: "7561bd04a5181b0283b33cea7d191cf8"
server_key_iv: "3a44482aac5ab60bccfd36c522bd81d1"
server_key_version: "encryptionkey@0"
I20260620 00:10:44.927179 20058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "bb06fda903414ed5afcc41c49a40908e"
format_stamp: "Formatted at 2026-06-20 00:10:44 on dist-test-slave-grqt"
server_key: "7561bd04a5181b0283b33cea7d191cf8"
server_key_iv: "3a44482aac5ab60bccfd36c522bd81d1"
server_key_version: "encryptionkey@0"
I20260620 00:10:44.928521 20058 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:10:44.929422 20072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:44.929628 20058 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:10:44.929725 20058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "bb06fda903414ed5afcc41c49a40908e"
format_stamp: "Formatted at 2026-06-20 00:10:44 on dist-test-slave-grqt"
server_key: "7561bd04a5181b0283b33cea7d191cf8"
server_key_iv: "3a44482aac5ab60bccfd36c522bd81d1"
server_key_version: "encryptionkey@0"
I20260620 00:10:44.929826 20058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:10:44.943558 20058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:10:44.943910 20058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:10:44.944077 20058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:10:44.948221 20058 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43079
I20260620 00:10:44.948257 20124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43079 every 8 connection(s)
I20260620 00:10:44.948608 20058 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:10:44.949592 20125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:10:44.951995 20125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb06fda903414ed5afcc41c49a40908e: Bootstrap starting.
I20260620 00:10:44.952603 20125 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb06fda903414ed5afcc41c49a40908e: Neither blocks nor log segments found. Creating new log.
I20260620 00:10:44.952924 20125 log.cc:824] T 00000000000000000000000000000000 P bb06fda903414ed5afcc41c49a40908e: Log is configured to *not* fsync() on all Append() calls
I20260620 00:10:44.953744 20125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb06fda903414ed5afcc41c49a40908e: No bootstrap required, opened a new log
I20260620 00:10:44.955374 20125 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb06fda903414ed5afcc41c49a40908e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb06fda903414ed5afcc41c49a40908e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43079 } }
I20260620 00:10:44.955542 20125 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb06fda903414ed5afcc41c49a40908e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:10:44.955590 20125 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb06fda903414ed5afcc41c49a40908e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb06fda903414ed5afcc41c49a40908e, State: Initialized, Role: FOLLOWER
I20260620 00:10:44.955771 20125 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb06fda903414ed5afcc41c49a40908e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb06fda903414ed5afcc41c49a40908e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43079 } }
I20260620 00:10:44.955858 20125 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb06fda903414ed5afcc41c49a40908e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:10:44.955899 20125 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb06fda903414ed5afcc41c49a40908e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:10:44.955950 20125 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb06fda903414ed5afcc41c49a40908e [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:10:44.956529 20125 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb06fda903414ed5afcc41c49a40908e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb06fda903414ed5afcc41c49a40908e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43079 } }
I20260620 00:10:44.956655 20125 leader_election.cc:302] T 00000000000000000000000000000000 P bb06fda903414ed5afcc41c49a40908e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb06fda903414ed5afcc41c49a40908e; no voters:
I20260620 00:10:44.956859 20125 leader_election.cc:288] T 00000000000000000000000000000000 P bb06fda903414ed5afcc41c49a40908e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:10:44.956915 20128 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb06fda903414ed5afcc41c49a40908e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:10:44.957141 20125 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb06fda903414ed5afcc41c49a40908e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:10:44.957187 20128 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb06fda903414ed5afcc41c49a40908e [term 1 LEADER]: Becoming Leader. State: Replica: bb06fda903414ed5afcc41c49a40908e, State: Running, Role: LEADER
I20260620 00:10:44.957314 20128 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb06fda903414ed5afcc41c49a40908e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb06fda903414ed5afcc41c49a40908e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43079 } }
I20260620 00:10:44.957770 20129 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb06fda903414ed5afcc41c49a40908e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb06fda903414ed5afcc41c49a40908e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb06fda903414ed5afcc41c49a40908e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43079 } } }
I20260620 00:10:44.957859 20130 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb06fda903414ed5afcc41c49a40908e [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb06fda903414ed5afcc41c49a40908e. Latest consensus state: current_term: 1 leader_uuid: "bb06fda903414ed5afcc41c49a40908e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb06fda903414ed5afcc41c49a40908e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43079 } } }
I20260620 00:10:44.957947 20130 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb06fda903414ed5afcc41c49a40908e [sys.catalog]: This master's current role is: LEADER
I20260620 00:10:44.957871 20129 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb06fda903414ed5afcc41c49a40908e [sys.catalog]: This master's current role is: LEADER
I20260620 00:10:44.958323 20137 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:10:44.958329 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 20058
I20260620 00:10:44.958420 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:10:44.958624 392 external_mini_cluster.cc:1428] Setting key 5f4b972e8f323128a99916c0573336d2
I20260620 00:10:44.958789 20137 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:10:44.960580 20137 catalog_manager.cc:1347] Generated new cluster ID: 977dc544689b463a8772d38de9147d34
I20260620 00:10:44.960644 20137 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:10:44.964859 20144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:44.959086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35526 (local address 127.0.98.62:43079)
0620 00:10:44.959365 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:44.959374 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:44.959499 (+ 125us) server_negotiation.cc:407] Connection header received
0620 00:10:44.959642 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:44.959649 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:44.959717 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:44.959864 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:44.960096 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:44.961440 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:44.962558 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:44.963217 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:44.963355 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:44.964093 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:44.964097 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:44.964097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:10:44.964323 (+ 226us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:44.964368 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:10:44.964383 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:44.964456 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:44.964541 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 00:10:44.964630 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":163,"threads_started":1}
I20260620 00:10:44.965487 20137 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:10:44.966018 20137 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:10:44.984524 20137 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb06fda903414ed5afcc41c49a40908e: Generated new TSK 0
I20260620 00:10:44.984761 20137 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:10:44.987088 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43079
--builtin_ntp_servers=127.0.98.20:38533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 00:10:45.069641 20149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:10:45.069844 20149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:10:45.069866 20149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:10:45.071213 20149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:10:45.071255 20149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:10:45.071282 20149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:10:45.072657 20149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:10:45.072904 20149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:10:45.073217 20149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:10:45.079234 20154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:45.079247 20155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:45.079595 20157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:45.079737 20149 server_base.cc:1034] running on GCE node
I20260620 00:10:45.079975 20149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:10:45.080262 20149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:10:45.081442 20149 hybrid_clock.cc:648] HybridClock initialized: now 1781914245081424 us; error 32 us; skew 500 ppm
I20260620 00:10:45.082939 20149 webserver.cc:460] Webserver started at http://127.0.98.1:34977/ using document root <none> and password file <none>
I20260620 00:10:45.083247 20149 fs_manager.cc:359] Metadata directory not provided
I20260620 00:10:45.083319 20149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:10:45.083480 20149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:10:45.084584 20149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-0/data/instance:
uuid: "26d8bd8533074d67bfdadebf0aa84b60"
format_stamp: "Formatted at 2026-06-20 00:10:45 on dist-test-slave-grqt"
server_key: "2acbb687c9dc40961a3f2812f5e01725"
server_key_iv: "e7b246daf2a0db1ab5dc90517d67a92f"
server_key_version: "encryptionkey@0"
I20260620 00:10:45.084965 20149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-0/wal/instance:
uuid: "26d8bd8533074d67bfdadebf0aa84b60"
format_stamp: "Formatted at 2026-06-20 00:10:45 on dist-test-slave-grqt"
server_key: "2acbb687c9dc40961a3f2812f5e01725"
server_key_iv: "e7b246daf2a0db1ab5dc90517d67a92f"
server_key_version: "encryptionkey@0"
I20260620 00:10:45.086391 20149 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:10:45.087111 20163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:45.087301 20149 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 00:10:45.087402 20149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-0/wal
uuid: "26d8bd8533074d67bfdadebf0aa84b60"
format_stamp: "Formatted at 2026-06-20 00:10:45 on dist-test-slave-grqt"
server_key: "2acbb687c9dc40961a3f2812f5e01725"
server_key_iv: "e7b246daf2a0db1ab5dc90517d67a92f"
server_key_version: "encryptionkey@0"
I20260620 00:10:45.087479 20149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:10:45.096117 20149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:10:45.096410 20149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:10:45.096561 20149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:10:45.096783 20149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:10:45.097116 20149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:10:45.097172 20149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:45.097211 20149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:10:45.097239 20149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:45.102937 20149 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32913
I20260620 00:10:45.102983 20276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32913 every 8 connection(s)
I20260620 00:10:45.103322 20149 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:10:45.109014 20144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:45.104169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58293 (local address 127.0.98.62:43079)
0620 00:10:45.104263 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:45.104266 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:45.104519 (+ 253us) server_negotiation.cc:407] Connection header received
0620 00:10:45.105001 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:45.105003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:45.105016 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:45.105074 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:45.105918 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:45.106515 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:45.107507 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:45.107749 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:45.107813 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:45.108712 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:45.108714 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:45.108714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:10:45.108719 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:45.108736 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:10:45.108740 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:45.108791 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:45.108897 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 00:10:45.108929 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 00:10:45.109248 20279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:45.104141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43079 (local address 127.0.98.1:58293)
0620 00:10:45.104359 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:45.104396 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:45.104883 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:45.105132 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:45.105135 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:45.105143 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:10:45.105807 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:45.105813 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:45.106607 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:45.106608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:45.107389 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:45.107392 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:45.107822 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:45.107823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:45.107957 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:45.108547 (+ 590us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:10:45.108547 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:10:45.108577 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:10:45.108601 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:10:45.108804 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:10:45.108808 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:10:45.108812 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:10:45.108900 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 00:10:45.109019 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":111,"threads_started":1}
I20260620 00:10:45.109895 20277 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43079
I20260620 00:10:45.109982 20277 heartbeater.cc:464] Registering TS with master...
I20260620 00:10:45.110170 20277 heartbeater.cc:511] Master 127.0.98.62:43079 requested a full tablet report, sending...
I20260620 00:10:45.110757 20089 ts_manager.cc:194] Registered new tserver with Master: 26d8bd8533074d67bfdadebf0aa84b60 (127.0.98.1:32913)
I20260620 00:10:45.111392 20089 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:58293
I20260620 00:10:45.111953 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 20149
I20260620 00:10:45.112067 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:10:45.112308 392 external_mini_cluster.cc:1428] Setting key 00e19cade3f66abc30150238dfca3d0f
I20260620 00:10:45.113322 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43079
--builtin_ntp_servers=127.0.98.20:38533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 00:10:45.196856 20280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:10:45.197079 20280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:10:45.197101 20280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:10:45.198388 20280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:10:45.198436 20280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:10:45.198462 20280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:10:45.199756 20280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:10:45.199949 20280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:10:45.200207 20280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 00:10:45.205530 20280 server_base.cc:1034] running on GCE node
W20260620 00:10:45.205487 20285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:45.205507 20288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:45.205503 20286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:45.206079 20280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:10:45.206358 20280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:10:45.207551 20280 hybrid_clock.cc:648] HybridClock initialized: now 1781914245207547 us; error 42 us; skew 500 ppm
I20260620 00:10:45.209033 20280 webserver.cc:460] Webserver started at http://127.0.98.2:37895/ using document root <none> and password file <none>
I20260620 00:10:45.209313 20280 fs_manager.cc:359] Metadata directory not provided
I20260620 00:10:45.209414 20280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:10:45.209571 20280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:10:45.210675 20280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-1/data/instance:
uuid: "174d3e594bd1430ebf6e6df70f137424"
format_stamp: "Formatted at 2026-06-20 00:10:45 on dist-test-slave-grqt"
server_key: "08b2c61400b7ea0d893de0c888fef981"
server_key_iv: "59f92b6be0865cf57859e0c36e6a2d00"
server_key_version: "encryptionkey@0"
I20260620 00:10:45.211048 20280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-1/wal/instance:
uuid: "174d3e594bd1430ebf6e6df70f137424"
format_stamp: "Formatted at 2026-06-20 00:10:45 on dist-test-slave-grqt"
server_key: "08b2c61400b7ea0d893de0c888fef981"
server_key_iv: "59f92b6be0865cf57859e0c36e6a2d00"
server_key_version: "encryptionkey@0"
I20260620 00:10:45.212649 20280 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 00:10:45.213502 20294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:45.213685 20280 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:10:45.213786 20280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-1/wal
uuid: "174d3e594bd1430ebf6e6df70f137424"
format_stamp: "Formatted at 2026-06-20 00:10:45 on dist-test-slave-grqt"
server_key: "08b2c61400b7ea0d893de0c888fef981"
server_key_iv: "59f92b6be0865cf57859e0c36e6a2d00"
server_key_version: "encryptionkey@0"
I20260620 00:10:45.213876 20280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:10:45.237260 20280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:10:45.237644 20280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:10:45.237808 20280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:10:45.238063 20280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:10:45.238471 20280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:10:45.238533 20280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:45.238577 20280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:10:45.238606 20280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:45.244910 20280 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35565
I20260620 00:10:45.244961 20407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35565 every 8 connection(s)
I20260620 00:10:45.245358 20280 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:10:45.248533 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 20280
I20260620 00:10:45.248628 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:10:45.248843 392 external_mini_cluster.cc:1428] Setting key 2298ec3e2a9dc027a317cae2a2d4d3ab
I20260620 00:10:45.249924 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43079
--builtin_ntp_servers=127.0.98.20:38533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 00:10:45.251451 20144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:45.246169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47609 (local address 127.0.98.62:43079)
0620 00:10:45.246234 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:45.246236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:45.246471 (+ 235us) server_negotiation.cc:407] Connection header received
0620 00:10:45.247031 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:45.247033 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:45.247045 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:45.247098 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:45.247799 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:45.248333 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:45.249624 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:45.249852 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:45.249953 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:45.251032 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:45.251033 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:45.251034 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:10:45.251039 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:45.251080 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:10:45.251086 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:45.251150 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:45.251297 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 00:10:45.251333 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 00:10:45.251611 20410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:45.246178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43079 (local address 127.0.98.2:47609)
0620 00:10:45.246352 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:45.246386 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:45.246899 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:45.247178 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:45.247182 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:45.247190 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:10:45.247700 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:45.247705 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:45.248430 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:45.248431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:45.249541 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:45.249545 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:45.249967 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:45.249969 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:45.250188 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:45.250823 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:10:45.250823 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:10:45.250855 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:10:45.250882 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:10:45.251181 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:10:45.251188 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:10:45.251192 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:10:45.251284 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 00:10:45.251399 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":72,"threads_started":1}
I20260620 00:10:45.252210 20408 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43079
I20260620 00:10:45.252341 20408 heartbeater.cc:464] Registering TS with master...
I20260620 00:10:45.252557 20408 heartbeater.cc:511] Master 127.0.98.62:43079 requested a full tablet report, sending...
I20260620 00:10:45.252930 20089 ts_manager.cc:194] Registered new tserver with Master: 174d3e594bd1430ebf6e6df70f137424 (127.0.98.2:35565)
I20260620 00:10:45.253543 20089 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:47609
W20260620 00:10:45.336973 20411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:10:45.337186 20411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:10:45.337208 20411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:10:45.338322 20411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:10:45.338361 20411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:10:45.338385 20411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:10:45.339668 20411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:10:45.339860 20411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:10:45.340116 20411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:10:45.345122 20419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:45.345132 20417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:45.345232 20416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:45.345332 20411 server_base.cc:1034] running on GCE node
I20260620 00:10:45.345676 20411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:10:45.345930 20411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:10:45.347115 20411 hybrid_clock.cc:648] HybridClock initialized: now 1781914245347094 us; error 41 us; skew 500 ppm
I20260620 00:10:45.348397 20411 webserver.cc:460] Webserver started at http://127.0.98.3:44411/ using document root <none> and password file <none>
I20260620 00:10:45.348640 20411 fs_manager.cc:359] Metadata directory not provided
I20260620 00:10:45.348708 20411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:10:45.348855 20411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:10:45.349989 20411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-2/data/instance:
uuid: "b2eb61db7faf4d09a1c76f6721af31d8"
format_stamp: "Formatted at 2026-06-20 00:10:45 on dist-test-slave-grqt"
server_key: "b22bdfeeae6767b6fb7388eb526c7255"
server_key_iv: "af3b97fa3b69bb1956544f1ce2e58420"
server_key_version: "encryptionkey@0"
I20260620 00:10:45.350354 20411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b2eb61db7faf4d09a1c76f6721af31d8"
format_stamp: "Formatted at 2026-06-20 00:10:45 on dist-test-slave-grqt"
server_key: "b22bdfeeae6767b6fb7388eb526c7255"
server_key_iv: "af3b97fa3b69bb1956544f1ce2e58420"
server_key_version: "encryptionkey@0"
I20260620 00:10:45.351635 20411 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:10:45.352393 20425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:45.352593 20411 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:10:45.352691 20411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-2/wal
uuid: "b2eb61db7faf4d09a1c76f6721af31d8"
format_stamp: "Formatted at 2026-06-20 00:10:45 on dist-test-slave-grqt"
server_key: "b22bdfeeae6767b6fb7388eb526c7255"
server_key_iv: "af3b97fa3b69bb1956544f1ce2e58420"
server_key_version: "encryptionkey@0"
I20260620 00:10:45.352763 20411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:10:45.365295 20411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:10:45.365659 20411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:10:45.365814 20411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:10:45.366039 20411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:10:45.366392 20411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:10:45.366443 20411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:45.366479 20411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:10:45.366505 20411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:45.372254 20411 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36787
I20260620 00:10:45.372323 20538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36787 every 8 connection(s)
I20260620 00:10:45.372665 20411 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:10:45.376147 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 20411
I20260620 00:10:45.376230 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:10:45.376451 392 external_mini_cluster.cc:1428] Setting key 9801f5c4844d4d9cd159a2c17846587f
I20260620 00:10:45.378283 20144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:45.373476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51561 (local address 127.0.98.62:43079)
0620 00:10:45.373521 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:45.373523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:45.373831 (+ 308us) server_negotiation.cc:407] Connection header received
0620 00:10:45.374369 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:45.374370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:45.374385 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:45.374437 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:45.375146 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:45.375668 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:45.376804 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:45.377020 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:45.377102 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:45.377987 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:45.377988 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:45.377988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:10:45.377993 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:45.378009 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:10:45.378013 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:45.378071 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:45.378173 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 00:10:45.378204 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 00:10:45.378499 20541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:45.373522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43079 (local address 127.0.98.3:51561)
0620 00:10:45.373700 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:45.373734 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:45.374253 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:45.374511 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:45.374515 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:45.374526 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:10:45.375042 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:45.375046 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:45.375760 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:45.375761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:45.376713 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:45.376716 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:45.377107 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:45.377108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:45.377245 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:45.377803 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:10:45.377804 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:10:45.377832 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:10:45.377856 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:10:45.378085 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:10:45.378090 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:10:45.378093 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:10:45.378177 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 00:10:45.378279 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
I20260620 00:10:45.378963 20539 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43079
I20260620 00:10:45.379065 20539 heartbeater.cc:464] Registering TS with master...
I20260620 00:10:45.379240 20539 heartbeater.cc:511] Master 127.0.98.62:43079 requested a full tablet report, sending...
I20260620 00:10:45.379585 20089 ts_manager.cc:194] Registered new tserver with Master: b2eb61db7faf4d09a1c76f6721af31d8 (127.0.98.3:36787)
I20260620 00:10:45.379941 20089 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:51561
I20260620 00:10:45.388237 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:10:45.392470 20144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:45.389654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35540 (local address 127.0.98.62:43079)
0620 00:10:45.389711 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:45.389713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:45.389853 (+ 140us) server_negotiation.cc:407] Connection header received
0620 00:10:45.389914 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:45.389914 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:45.389925 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:45.390001 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:45.390142 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:45.390702 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:45.391463 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:45.391651 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:45.391715 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:45.392299 (+ 584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:45.392299 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:45.392300 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:10:45.392304 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:45.392314 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:10:45.392318 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:45.392358 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:45.392391 (+ 33us) server_negotiation.cc:299] Negotiation successful
0620 00:10:45.392411 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 00:10:45.394263 20089 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35540:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:10:45.395851 20089 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:10:45.402251 20550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:45.398257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36787 (local address 127.0.0.1:42674)
0620 00:10:45.398469 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:45.398479 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:45.398516 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:45.398834 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:45.398835 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:45.398838 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:10:45.398876 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:45.398878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:45.399878 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:45.399879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:45.401006 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:45.401009 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:45.401954 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:45.401955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:45.402038 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:45.402052 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 00:10:45.402127 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 00:10:45.402178 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":65,"threads_started":1}
I20260620 00:10:45.402262 20552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:45.398458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42674 (local address 127.0.98.3:36787)
0620 00:10:45.398665 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:45.398671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:45.398681 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:10:45.398721 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:45.398723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:45.398738 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:45.398820 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:10:45.399042 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:45.399790 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:45.401101 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:45.401841 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:45.401953 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:45.402048 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:45.402140 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 00:10:45.402177 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":100,"threads_started":1}
I20260620 00:10:45.402627 20547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:45.397993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35565 (local address 127.0.0.1:57796)
0620 00:10:45.398180 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:45.398222 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:45.398298 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:45.398980 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:45.398981 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:45.398982 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:10:45.399011 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:45.399012 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:45.400379 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:45.400380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:45.401448 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:45.401451 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:45.402393 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:45.402394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:45.402465 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:45.402469 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:10:45.402516 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 00:10:45.402565 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":111,"threads_started":1}
I20260620 00:10:45.402688 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:45.398207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57796 (local address 127.0.98.2:35565)
0620 00:10:45.398673 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:45.398677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:45.398687 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:10:45.398722 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:45.398724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:45.398738 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:45.398820 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:10:45.399095 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:45.400287 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:45.401530 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:45.402308 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:45.402373 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:45.402452 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:45.402545 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 00:10:45.402592 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":94,"threads_started":1}
I20260620 00:10:45.402768 20551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:45.397993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32913 (local address 127.0.0.1:46794)
0620 00:10:45.398377 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:45.398387 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:45.398437 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:45.398723 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:45.398726 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:45.398728 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:10:45.398778 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:45.398779 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:45.400268 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:45.400269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:45.401266 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:45.401269 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:45.402437 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:45.402438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:45.402513 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:45.402518 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:10:45.402573 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 00:10:45.402618 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":4,"thread_start_us":328,"threads_started":1}
I20260620 00:10:45.402823 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:45.398113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46794 (local address 127.0.98.1:32913)
0620 00:10:45.398431 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:45.398436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:45.398448 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:10:45.398524 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:45.398527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:45.398545 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:45.398630 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:10:45.398919 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:45.400165 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:45.401356 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:45.402223 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:45.402302 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:45.402396 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:45.402672 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 00:10:45.402724 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":97,"threads_started":1}
I20260620 00:10:45.403456 20342 tablet_service.cc:1467] Processing CreateTablet for tablet 00cca163680744069e2989b20f077c49 (DEFAULT_TABLE table=test-table [id=e7c51d0065614919a3c516fc8904e307]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:10:45.403491 20211 tablet_service.cc:1467] Processing CreateTablet for tablet 00cca163680744069e2989b20f077c49 (DEFAULT_TABLE table=test-table [id=e7c51d0065614919a3c516fc8904e307]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:10:45.403864 20211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00cca163680744069e2989b20f077c49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:10:45.403864 20342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00cca163680744069e2989b20f077c49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:10:45.405865 20473 tablet_service.cc:1467] Processing CreateTablet for tablet 00cca163680744069e2989b20f077c49 (DEFAULT_TABLE table=test-table [id=e7c51d0065614919a3c516fc8904e307]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:10:45.406250 20473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00cca163680744069e2989b20f077c49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:10:45.407670 20553 tablet_bootstrap.cc:492] T 00cca163680744069e2989b20f077c49 P 26d8bd8533074d67bfdadebf0aa84b60: Bootstrap starting.
I20260620 00:10:45.408365 20553 tablet_bootstrap.cc:654] T 00cca163680744069e2989b20f077c49 P 26d8bd8533074d67bfdadebf0aa84b60: Neither blocks nor log segments found. Creating new log.
I20260620 00:10:45.408675 20553 log.cc:824] T 00cca163680744069e2989b20f077c49 P 26d8bd8533074d67bfdadebf0aa84b60: Log is configured to *not* fsync() on all Append() calls
I20260620 00:10:45.409042 20554 tablet_bootstrap.cc:492] T 00cca163680744069e2989b20f077c49 P 174d3e594bd1430ebf6e6df70f137424: Bootstrap starting.
I20260620 00:10:45.409215 20556 tablet_bootstrap.cc:492] T 00cca163680744069e2989b20f077c49 P b2eb61db7faf4d09a1c76f6721af31d8: Bootstrap starting.
I20260620 00:10:45.409588 20553 tablet_bootstrap.cc:492] T 00cca163680744069e2989b20f077c49 P 26d8bd8533074d67bfdadebf0aa84b60: No bootstrap required, opened a new log
I20260620 00:10:45.409668 20553 ts_tablet_manager.cc:1397] T 00cca163680744069e2989b20f077c49 P 26d8bd8533074d67bfdadebf0aa84b60: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:10:45.409930 20554 tablet_bootstrap.cc:654] T 00cca163680744069e2989b20f077c49 P 174d3e594bd1430ebf6e6df70f137424: Neither blocks nor log segments found. Creating new log.
I20260620 00:10:45.409989 20556 tablet_bootstrap.cc:654] T 00cca163680744069e2989b20f077c49 P b2eb61db7faf4d09a1c76f6721af31d8: Neither blocks nor log segments found. Creating new log.
I20260620 00:10:45.410241 20554 log.cc:824] T 00cca163680744069e2989b20f077c49 P 174d3e594bd1430ebf6e6df70f137424: Log is configured to *not* fsync() on all Append() calls
I20260620 00:10:45.410362 20556 log.cc:824] T 00cca163680744069e2989b20f077c49 P b2eb61db7faf4d09a1c76f6721af31d8: Log is configured to *not* fsync() on all Append() calls
I20260620 00:10:45.411074 20556 tablet_bootstrap.cc:492] T 00cca163680744069e2989b20f077c49 P b2eb61db7faf4d09a1c76f6721af31d8: No bootstrap required, opened a new log
I20260620 00:10:45.411080 20554 tablet_bootstrap.cc:492] T 00cca163680744069e2989b20f077c49 P 174d3e594bd1430ebf6e6df70f137424: No bootstrap required, opened a new log
I20260620 00:10:45.411167 20554 ts_tablet_manager.cc:1397] T 00cca163680744069e2989b20f077c49 P 174d3e594bd1430ebf6e6df70f137424: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:10:45.411167 20556 ts_tablet_manager.cc:1397] T 00cca163680744069e2989b20f077c49 P b2eb61db7faf4d09a1c76f6721af31d8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:10:45.412606 20553 raft_consensus.cc:348] T 00cca163680744069e2989b20f077c49 P 26d8bd8533074d67bfdadebf0aa84b60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174d3e594bd1430ebf6e6df70f137424" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35565 } } peers { permanent_uuid: "26d8bd8533074d67bfdadebf0aa84b60" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32913 } } peers { permanent_uuid: "b2eb61db7faf4d09a1c76f6721af31d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36787 } }
I20260620 00:10:45.412801 20553 raft_consensus.cc:374] T 00cca163680744069e2989b20f077c49 P 26d8bd8533074d67bfdadebf0aa84b60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:10:45.412840 20553 raft_consensus.cc:729] T 00cca163680744069e2989b20f077c49 P 26d8bd8533074d67bfdadebf0aa84b60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d8bd8533074d67bfdadebf0aa84b60, State: Initialized, Role: FOLLOWER
I20260620 00:10:45.412997 20553 consensus_queue.cc:260] T 00cca163680744069e2989b20f077c49 P 26d8bd8533074d67bfdadebf0aa84b60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174d3e594bd1430ebf6e6df70f137424" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35565 } } peers { permanent_uuid: "26d8bd8533074d67bfdadebf0aa84b60" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32913 } } peers { permanent_uuid: "b2eb61db7faf4d09a1c76f6721af31d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36787 } }
I20260620 00:10:45.412988 20556 raft_consensus.cc:348] T 00cca163680744069e2989b20f077c49 P b2eb61db7faf4d09a1c76f6721af31d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174d3e594bd1430ebf6e6df70f137424" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35565 } } peers { permanent_uuid: "26d8bd8533074d67bfdadebf0aa84b60" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32913 } } peers { permanent_uuid: "b2eb61db7faf4d09a1c76f6721af31d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36787 } }
I20260620 00:10:45.413163 20556 raft_consensus.cc:374] T 00cca163680744069e2989b20f077c49 P b2eb61db7faf4d09a1c76f6721af31d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:10:45.413197 20556 raft_consensus.cc:729] T 00cca163680744069e2989b20f077c49 P b2eb61db7faf4d09a1c76f6721af31d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2eb61db7faf4d09a1c76f6721af31d8, State: Initialized, Role: FOLLOWER
I20260620 00:10:45.413290 20553 ts_tablet_manager.cc:1428] T 00cca163680744069e2989b20f077c49 P 26d8bd8533074d67bfdadebf0aa84b60: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 00:10:45.413316 20556 consensus_queue.cc:260] T 00cca163680744069e2989b20f077c49 P b2eb61db7faf4d09a1c76f6721af31d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174d3e594bd1430ebf6e6df70f137424" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35565 } } peers { permanent_uuid: "26d8bd8533074d67bfdadebf0aa84b60" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32913 } } peers { permanent_uuid: "b2eb61db7faf4d09a1c76f6721af31d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36787 } }
I20260620 00:10:45.413451 20277 heartbeater.cc:503] Master 127.0.98.62:43079 was elected leader, sending a full tablet report...
I20260620 00:10:45.413595 20556 ts_tablet_manager.cc:1428] T 00cca163680744069e2989b20f077c49 P b2eb61db7faf4d09a1c76f6721af31d8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 00:10:45.413640 20539 heartbeater.cc:503] Master 127.0.98.62:43079 was elected leader, sending a full tablet report...
I20260620 00:10:45.414029 20554 raft_consensus.cc:348] T 00cca163680744069e2989b20f077c49 P 174d3e594bd1430ebf6e6df70f137424 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174d3e594bd1430ebf6e6df70f137424" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35565 } } peers { permanent_uuid: "26d8bd8533074d67bfdadebf0aa84b60" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32913 } } peers { permanent_uuid: "b2eb61db7faf4d09a1c76f6721af31d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36787 } }
I20260620 00:10:45.414160 20554 raft_consensus.cc:374] T 00cca163680744069e2989b20f077c49 P 174d3e594bd1430ebf6e6df70f137424 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:10:45.414188 20554 raft_consensus.cc:729] T 00cca163680744069e2989b20f077c49 P 174d3e594bd1430ebf6e6df70f137424 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 174d3e594bd1430ebf6e6df70f137424, State: Initialized, Role: FOLLOWER
I20260620 00:10:45.414351 20554 consensus_queue.cc:260] T 00cca163680744069e2989b20f077c49 P 174d3e594bd1430ebf6e6df70f137424 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174d3e594bd1430ebf6e6df70f137424" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35565 } } peers { permanent_uuid: "26d8bd8533074d67bfdadebf0aa84b60" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32913 } } peers { permanent_uuid: "b2eb61db7faf4d09a1c76f6721af31d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36787 } }
I20260620 00:10:45.414635 20554 ts_tablet_manager.cc:1428] T 00cca163680744069e2989b20f077c49 P 174d3e594bd1430ebf6e6df70f137424: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260620 00:10:45.414686 20408 heartbeater.cc:503] Master 127.0.98.62:43079 was elected leader, sending a full tablet report...
W20260620 00:10:45.495954 20409 tablet.cc:2369] T 00cca163680744069e2989b20f077c49 P 174d3e594bd1430ebf6e6df70f137424: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:10:45.598869 20559 raft_consensus.cc:482] T 00cca163680744069e2989b20f077c49 P 26d8bd8533074d67bfdadebf0aa84b60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:10:45.599010 20559 raft_consensus.cc:504] T 00cca163680744069e2989b20f077c49 P 26d8bd8533074d67bfdadebf0aa84b60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174d3e594bd1430ebf6e6df70f137424" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35565 } } peers { permanent_uuid: "26d8bd8533074d67bfdadebf0aa84b60" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32913 } } peers { permanent_uuid: "b2eb61db7faf4d09a1c76f6721af31d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36787 } }
I20260620 00:10:45.599323 20559 leader_election.cc:288] T 00cca163680744069e2989b20f077c49 P 26d8bd8533074d67bfdadebf0aa84b60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 174d3e594bd1430ebf6e6df70f137424 (127.0.98.2:35565), b2eb61db7faf4d09a1c76f6721af31d8 (127.0.98.3:36787)
I20260620 00:10:45.602293 20279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:45.599543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35565 (local address 127.0.98.1:46697)
0620 00:10:45.599608 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:45.599621 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:45.599670 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:45.599850 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:45.599852 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:45.599853 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:10:45.599931 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:45.599932 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:45.600587 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:45.600588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:45.601354 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:45.601355 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:45.602012 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:45.602013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:45.602084 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:45.602087 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:10:45.602156 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 00:10:45.602196 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":5}
I20260620 00:10:45.602330 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:45.599636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46697 (local address 127.0.98.2:35565)
0620 00:10:45.599695 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:45.599697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:45.599710 (+ 13us) server_negotiation.cc:407] Connection header received
0620 00:10:45.599744 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:45.599745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:45.599758 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:45.599835 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:10:45.600012 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:45.600499 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:45.601489 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:45.601941 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:45.601996 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:45.602005 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:45.602190 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 00:10:45.602230 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 00:10:45.602593 20562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:45.599669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36787 (local address 127.0.98.1:32991)
0620 00:10:45.599763 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:45.599771 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 00:10:45.599835 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:10:45.599988 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:10:45.599989 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:10:45.599991 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:10:45.600080 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:45.600081 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:45.600714 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:45.600715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:45.601540 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:10:45.601543 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:10:45.602307 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:10:45.602308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:10:45.602402 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:45.602406 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:10:45.602477 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 00:10:45.602523 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":64,"threads_started":1}
I20260620 00:10:45.602602 20552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:45.599784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32991 (local address 127.0.98.3:36787)
0620 00:10:45.599854 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:45.599857 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:45.599867 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:10:45.599898 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:45.599899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:45.599907 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:45.599983 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:10:45.600164 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:45.600621 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:45.601626 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:45.602208 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:45.602274 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:45.602282 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:45.602493 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 00:10:45.602542 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 00:10:45.602674 20362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00cca163680744069e2989b20f077c49" candidate_uuid: "26d8bd8533074d67bfdadebf0aa84b60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "174d3e594bd1430ebf6e6df70f137424" is_pre_election: true
I20260620 00:10:45.602849 20362 raft_consensus.cc:2454] T 00cca163680744069e2989b20f077c49 P 174d3e594bd1430ebf6e6df70f137424 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26d8bd8533074d67bfdadebf0aa84b60 in term 0.
I20260620 00:10:45.603155 20164 leader_election.cc:302] T 00cca163680744069e2989b20f077c49 P 26d8bd8533074d67bfdadebf0aa84b60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 174d3e594bd1430ebf6e6df70f137424, 26d8bd8533074d67bfdadebf0aa84b60; no voters:
I20260620 00:10:45.603302 20559 raft_consensus.cc:2781] T 00cca163680744069e2989b20f077c49 P 26d8bd8533074d67bfdadebf0aa84b60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:10:45.603248 20493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00cca163680744069e2989b20f077c49" candidate_uuid: "26d8bd8533074d67bfdadebf0aa84b60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2eb61db7faf4d09a1c76f6721af31d8" is_pre_election: true
I20260620 00:10:45.603368 20559 raft_consensus.cc:482] T 00cca163680744069e2989b20f077c49 P 26d8bd8533074d67bfdadebf0aa84b60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:10:45.603404 20559 raft_consensus.cc:3037] T 00cca163680744069e2989b20f077c49 P 26d8bd8533074d67bfdadebf0aa84b60 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:10:45.603433 20493 raft_consensus.cc:2454] T 00cca163680744069e2989b20f077c49 P b2eb61db7faf4d09a1c76f6721af31d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26d8bd8533074d67bfdadebf0aa84b60 in term 0.
W20260620 00:10:45.603999 20278 tablet.cc:2369] T 00cca163680744069e2989b20f077c49 P 26d8bd8533074d67bfdadebf0aa84b60: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:10:45.604282 20559 raft_consensus.cc:504] T 00cca163680744069e2989b20f077c49 P 26d8bd8533074d67bfdadebf0aa84b60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174d3e594bd1430ebf6e6df70f137424" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35565 } } peers { permanent_uuid: "26d8bd8533074d67bfdadebf0aa84b60" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32913 } } peers { permanent_uuid: "b2eb61db7faf4d09a1c76f6721af31d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36787 } }
I20260620 00:10:45.604408 20559 leader_election.cc:288] T 00cca163680744069e2989b20f077c49 P 26d8bd8533074d67bfdadebf0aa84b60 [CANDIDATE]: Term 1 election: Requested vote from peers 174d3e594bd1430ebf6e6df70f137424 (127.0.98.2:35565), b2eb61db7faf4d09a1c76f6721af31d8 (127.0.98.3:36787)
I20260620 00:10:45.604562 20493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00cca163680744069e2989b20f077c49" candidate_uuid: "26d8bd8533074d67bfdadebf0aa84b60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2eb61db7faf4d09a1c76f6721af31d8"
I20260620 00:10:45.604575 20362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00cca163680744069e2989b20f077c49" candidate_uuid: "26d8bd8533074d67bfdadebf0aa84b60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "174d3e594bd1430ebf6e6df70f137424"
I20260620 00:10:45.604642 20493 raft_consensus.cc:3037] T 00cca163680744069e2989b20f077c49 P b2eb61db7faf4d09a1c76f6721af31d8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:10:45.604676 20362 raft_consensus.cc:3037] T 00cca163680744069e2989b20f077c49 P 174d3e594bd1430ebf6e6df70f137424 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:10:45.605445 20493 raft_consensus.cc:2454] T 00cca163680744069e2989b20f077c49 P b2eb61db7faf4d09a1c76f6721af31d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26d8bd8533074d67bfdadebf0aa84b60 in term 1.
I20260620 00:10:45.605463 20362 raft_consensus.cc:2454] T 00cca163680744069e2989b20f077c49 P 174d3e594bd1430ebf6e6df70f137424 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26d8bd8533074d67bfdadebf0aa84b60 in term 1.
I20260620 00:10:45.605603 20164 leader_election.cc:302] T 00cca163680744069e2989b20f077c49 P 26d8bd8533074d67bfdadebf0aa84b60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26d8bd8533074d67bfdadebf0aa84b60, b2eb61db7faf4d09a1c76f6721af31d8; no voters:
I20260620 00:10:45.605711 20559 raft_consensus.cc:2781] T 00cca163680744069e2989b20f077c49 P 26d8bd8533074d67bfdadebf0aa84b60 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:10:45.605844 20559 raft_consensus.cc:686] T 00cca163680744069e2989b20f077c49 P 26d8bd8533074d67bfdadebf0aa84b60 [term 1 LEADER]: Becoming Leader. State: Replica: 26d8bd8533074d67bfdadebf0aa84b60, State: Running, Role: LEADER
I20260620 00:10:45.605937 20559 consensus_queue.cc:237] T 00cca163680744069e2989b20f077c49 P 26d8bd8533074d67bfdadebf0aa84b60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174d3e594bd1430ebf6e6df70f137424" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35565 } } peers { permanent_uuid: "26d8bd8533074d67bfdadebf0aa84b60" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32913 } } peers { permanent_uuid: "b2eb61db7faf4d09a1c76f6721af31d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36787 } }
I20260620 00:10:45.606761 20089 catalog_manager.cc:5495] T 00cca163680744069e2989b20f077c49 P 26d8bd8533074d67bfdadebf0aa84b60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26d8bd8533074d67bfdadebf0aa84b60 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "26d8bd8533074d67bfdadebf0aa84b60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174d3e594bd1430ebf6e6df70f137424" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26d8bd8533074d67bfdadebf0aa84b60" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2eb61db7faf4d09a1c76f6721af31d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36787 } health_report { overall_health: UNKNOWN } } }
W20260620 00:10:45.623242 20540 tablet.cc:2369] T 00cca163680744069e2989b20f077c49 P b2eb61db7faf4d09a1c76f6721af31d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:10:45.628671 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:45.625929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46800 (local address 127.0.98.1:32913)
0620 00:10:45.626010 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:45.626013 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:45.626022 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:10:45.626068 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:45.626069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:45.626089 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:45.626175 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:10:45.626294 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:45.626878 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:45.627718 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:45.627948 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:45.628052 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:45.628252 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:10:45.628367 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:10:45.628483 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:45.628516 (+ 33us) server_negotiation.cc:299] Negotiation successful
0620 00:10:45.628540 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 00:10:45.630645 20493 raft_consensus.cc:1261] T 00cca163680744069e2989b20f077c49 P b2eb61db7faf4d09a1c76f6721af31d8 [term 1 FOLLOWER]: Refusing update from remote peer 26d8bd8533074d67bfdadebf0aa84b60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:10:45.630684 20362 raft_consensus.cc:1261] T 00cca163680744069e2989b20f077c49 P 174d3e594bd1430ebf6e6df70f137424 [term 1 FOLLOWER]: Refusing update from remote peer 26d8bd8533074d67bfdadebf0aa84b60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:10:45.631043 20559 consensus_queue.cc:1035] T 00cca163680744069e2989b20f077c49 P 26d8bd8533074d67bfdadebf0aa84b60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2eb61db7faf4d09a1c76f6721af31d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:10:45.631211 20563 consensus_queue.cc:1035] T 00cca163680744069e2989b20f077c49 P 26d8bd8533074d67bfdadebf0aa84b60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "174d3e594bd1430ebf6e6df70f137424" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:10:45.634487 20566 mvcc.cc:205] Tried to move back new op lower bound from 7298720750100144128 to 7298720750002499584. Current Snapshot: MvccSnapshot[applied={T|T < 7298720750100144128}]
I20260620 00:10:45.634996 20567 mvcc.cc:205] Tried to move back new op lower bound from 7298720750100144128 to 7298720750002499584. Current Snapshot: MvccSnapshot[applied={T|T < 7298720750100144128}]
I20260620 00:10:45.636642 20088 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:10:45.636751 20088 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:10:45.637686 20088 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb06fda903414ed5afcc41c49a40908e: Sending DeleteTablet for 3 replicas of tablet 00cca163680744069e2989b20f077c49
I20260620 00:10:45.638072 20473 tablet_service.cc:1514] Processing DeleteTablet for tablet 00cca163680744069e2989b20f077c49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:10:45 UTC) from {username='slave'} at 127.0.0.1:42674
I20260620 00:10:45.638099 20342 tablet_service.cc:1514] Processing DeleteTablet for tablet 00cca163680744069e2989b20f077c49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:10:45 UTC) from {username='slave'} at 127.0.0.1:57796
I20260620 00:10:45.638321 20576 tablet_replica.cc:330] T 00cca163680744069e2989b20f077c49 P 174d3e594bd1430ebf6e6df70f137424: stopping tablet replica
I20260620 00:10:45.638341 20575 tablet_replica.cc:330] T 00cca163680744069e2989b20f077c49 P b2eb61db7faf4d09a1c76f6721af31d8: stopping tablet replica
I20260620 00:10:45.638319 20211 tablet_service.cc:1514] Processing DeleteTablet for tablet 00cca163680744069e2989b20f077c49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:10:45 UTC) from {username='slave'} at 127.0.0.1:46794
I20260620 00:10:45.638434 20575 raft_consensus.cc:2229] T 00cca163680744069e2989b20f077c49 P b2eb61db7faf4d09a1c76f6721af31d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:10:45.638548 20576 raft_consensus.cc:2229] T 00cca163680744069e2989b20f077c49 P 174d3e594bd1430ebf6e6df70f137424 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:10:45.638562 20575 raft_consensus.cc:2258] T 00cca163680744069e2989b20f077c49 P b2eb61db7faf4d09a1c76f6721af31d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:10:45.638722 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 20149
I20260620 00:10:45.639904 20575 ts_tablet_manager.cc:1905] T 00cca163680744069e2989b20f077c49 P b2eb61db7faf4d09a1c76f6721af31d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:10:45.641906 20575 ts_tablet_manager.cc:1918] T 00cca163680744069e2989b20f077c49 P b2eb61db7faf4d09a1c76f6721af31d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:10:45.641993 20575 log.cc:1196] T 00cca163680744069e2989b20f077c49 P b2eb61db7faf4d09a1c76f6721af31d8: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-2/wal/wals/00cca163680744069e2989b20f077c49
I20260620 00:10:45.642246 20575 ts_tablet_manager.cc:1939] T 00cca163680744069e2989b20f077c49 P b2eb61db7faf4d09a1c76f6721af31d8: Deleting consensus metadata
I20260620 00:10:45.638715 20576 raft_consensus.cc:2258] T 00cca163680744069e2989b20f077c49 P 174d3e594bd1430ebf6e6df70f137424 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:10:45.642844 20576 ts_tablet_manager.cc:1905] T 00cca163680744069e2989b20f077c49 P 174d3e594bd1430ebf6e6df70f137424: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:10:45.642871 20073 catalog_manager.cc:4841] TS b2eb61db7faf4d09a1c76f6721af31d8 (127.0.98.3:36787): tablet 00cca163680744069e2989b20f077c49 (table test-table [id=e7c51d0065614919a3c516fc8904e307]) successfully deleted
I20260620 00:10:45.644129 20576 ts_tablet_manager.cc:1918] T 00cca163680744069e2989b20f077c49 P 174d3e594bd1430ebf6e6df70f137424: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:10:45.644214 20576 log.cc:1196] T 00cca163680744069e2989b20f077c49 P 174d3e594bd1430ebf6e6df70f137424: Deleting WAL directory at /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914001714546-392-0/minicluster-data/ts-1/wal/wals/00cca163680744069e2989b20f077c49
I20260620 00:10:45.644433 20576 ts_tablet_manager.cc:1939] T 00cca163680744069e2989b20f077c49 P 174d3e594bd1430ebf6e6df70f137424: Deleting consensus metadata
I20260620 00:10:45.644836 20073 catalog_manager.cc:4841] TS 174d3e594bd1430ebf6e6df70f137424 (127.0.98.2:35565): tablet 00cca163680744069e2989b20f077c49 (table test-table [id=e7c51d0065614919a3c516fc8904e307]) successfully deleted
W20260620 00:10:45.645612 20076 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:32913 (error 108)
I20260620 00:10:45.646112 20551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:45.645965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32913 (local address 127.0.0.1:46814)
0620 00:10:45.646028 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 00:10:45.646075 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 00:10:45.646111 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 20280
W20260620 00:10:45.648017 20076 catalog_manager.cc:4584] TS 26d8bd8533074d67bfdadebf0aa84b60 (127.0.98.1:32913): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 00cca163680744069e2989b20f077c49: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32913: connect: Connection refused (error 111)
I20260620 00:10:45.650689 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 20411
I20260620 00:10:45.654739 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 20058
2026-06-20T00:10:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (848 ms)
[----------] 6 tests from EncryptionPolicyTest (8681 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:10:45 dist-test-slave-grqt krb5kdc[20581](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:10:45 dist-test-slave-grqt krb5kdc[20581](info): set up 2 sockets
Jun 20 00:10:45 dist-test-slave-grqt krb5kdc[20581](info): commencing operation
krb5kdc: starting...
W20260620 00:10:46.697422 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:10:46 dist-test-slave-grqt krb5kdc[20581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:10:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:10:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 00:10:46.817101 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35835
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:10:46.899623 20594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:10:46.899904 20594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:10:46.899948 20594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:10:46.901199 20594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:10:46.901284 20594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:10:46.901311 20594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:10:46.901345 20594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:10:46.901412 20594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:10:46.903030 20594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35835
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:10:46.903349 20594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:10:46.903612 20594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:10:46.909031 20599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:46.909024 20600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:46.909017 20602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:46.909101 20594 server_base.cc:1034] running on GCE node
I20260620 00:10:46.909578 20594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:10:46.909968 20594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:10:46.911203 20594 hybrid_clock.cc:648] HybridClock initialized: now 1781914246911158 us; error 65 us; skew 500 ppm
Jun 20 00:10:46 dist-test-slave-grqt krb5kdc[20581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:10:46.914104 20594 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 00:10:46.914604 20594 webserver.cc:460] Webserver started at http://127.0.0.1:41559/ using document root <none> and password file <none>
I20260620 00:10:46.914855 20594 fs_manager.cc:359] Metadata directory not provided
I20260620 00:10:46.914929 20594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:10:46.915093 20594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:10:46.916226 20594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "2195e0b5078245daa2ffa1a4df4ebdba"
format_stamp: "Formatted at 2026-06-20 00:10:46 on dist-test-slave-grqt"
server_key: "12fee065200ad49427abcfd751571f28"
server_key_iv: "51e6baf02e0beeccba91c619293188b6"
server_key_version: "encryptionkey@0"
I20260620 00:10:46.916603 20594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "2195e0b5078245daa2ffa1a4df4ebdba"
format_stamp: "Formatted at 2026-06-20 00:10:46 on dist-test-slave-grqt"
server_key: "12fee065200ad49427abcfd751571f28"
server_key_iv: "51e6baf02e0beeccba91c619293188b6"
server_key_version: "encryptionkey@0"
I20260620 00:10:46.918206 20594 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:10:46.919162 20609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:46.919407 20594 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:10:46.919559 20594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "2195e0b5078245daa2ffa1a4df4ebdba"
format_stamp: "Formatted at 2026-06-20 00:10:46 on dist-test-slave-grqt"
server_key: "12fee065200ad49427abcfd751571f28"
server_key_iv: "51e6baf02e0beeccba91c619293188b6"
server_key_version: "encryptionkey@0"
I20260620 00:10:46.919651 20594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:10:46.937695 20594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:10:46.939870 20594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:10:46.940039 20594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:10:46.944057 20594 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35835
I20260620 00:10:46.944116 20661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35835 every 8 connection(s)
I20260620 00:10:46.944489 20594 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:10:46.945410 20662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:10:46.947696 20662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2195e0b5078245daa2ffa1a4df4ebdba: Bootstrap starting.
I20260620 00:10:46.948244 20662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2195e0b5078245daa2ffa1a4df4ebdba: Neither blocks nor log segments found. Creating new log.
I20260620 00:10:46.948482 20662 log.cc:824] T 00000000000000000000000000000000 P 2195e0b5078245daa2ffa1a4df4ebdba: Log is configured to *not* fsync() on all Append() calls
I20260620 00:10:46.949167 20662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2195e0b5078245daa2ffa1a4df4ebdba: No bootstrap required, opened a new log
I20260620 00:10:46.950794 20662 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2195e0b5078245daa2ffa1a4df4ebdba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2195e0b5078245daa2ffa1a4df4ebdba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35835 } }
I20260620 00:10:46.950940 20662 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2195e0b5078245daa2ffa1a4df4ebdba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:10:46.950963 20662 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2195e0b5078245daa2ffa1a4df4ebdba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2195e0b5078245daa2ffa1a4df4ebdba, State: Initialized, Role: FOLLOWER
I20260620 00:10:46.951117 20662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2195e0b5078245daa2ffa1a4df4ebdba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2195e0b5078245daa2ffa1a4df4ebdba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35835 } }
I20260620 00:10:46.951195 20662 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2195e0b5078245daa2ffa1a4df4ebdba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:10:46.951217 20662 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2195e0b5078245daa2ffa1a4df4ebdba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:10:46.951246 20662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2195e0b5078245daa2ffa1a4df4ebdba [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:10:46.951761 20662 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2195e0b5078245daa2ffa1a4df4ebdba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2195e0b5078245daa2ffa1a4df4ebdba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35835 } }
I20260620 00:10:46.951848 20662 leader_election.cc:302] T 00000000000000000000000000000000 P 2195e0b5078245daa2ffa1a4df4ebdba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2195e0b5078245daa2ffa1a4df4ebdba; no voters:
I20260620 00:10:46.951974 20662 leader_election.cc:288] T 00000000000000000000000000000000 P 2195e0b5078245daa2ffa1a4df4ebdba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:10:46.952097 20665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2195e0b5078245daa2ffa1a4df4ebdba [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:10:46.952275 20662 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2195e0b5078245daa2ffa1a4df4ebdba [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:10:46.952296 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 20594
I20260620 00:10:46.952322 20665 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2195e0b5078245daa2ffa1a4df4ebdba [term 1 LEADER]: Becoming Leader. State: Replica: 2195e0b5078245daa2ffa1a4df4ebdba, State: Running, Role: LEADER
I20260620 00:10:46.952411 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:10:46.952502 20665 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2195e0b5078245daa2ffa1a4df4ebdba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2195e0b5078245daa2ffa1a4df4ebdba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35835 } }
I20260620 00:10:46.952630 392 external_mini_cluster.cc:1428] Setting key 38d4ca4f0a20febe0d81e5fd7b7d3502
I20260620 00:10:46.952942 20666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2195e0b5078245daa2ffa1a4df4ebdba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2195e0b5078245daa2ffa1a4df4ebdba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2195e0b5078245daa2ffa1a4df4ebdba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35835 } } }
I20260620 00:10:46.953060 20666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2195e0b5078245daa2ffa1a4df4ebdba [sys.catalog]: This master's current role is: LEADER
I20260620 00:10:46.953305 20670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:10:46.953799 20667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2195e0b5078245daa2ffa1a4df4ebdba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2195e0b5078245daa2ffa1a4df4ebdba. Latest consensus state: current_term: 1 leader_uuid: "2195e0b5078245daa2ffa1a4df4ebdba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2195e0b5078245daa2ffa1a4df4ebdba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35835 } } }
I20260620 00:10:46.953964 20667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2195e0b5078245daa2ffa1a4df4ebdba [sys.catalog]: This master's current role is: LEADER
I20260620 00:10:46.954062 20670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:10:46.956190 20670 catalog_manager.cc:1347] Generated new cluster ID: 1e806ff990844ead9a5c41c15da7845e
I20260620 00:10:46.956245 20670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 00:10:46 dist-test-slave-grqt krb5kdc[20581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 00:10:46.964458 20670 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:10:46.965130 20670 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:10:46.965212 20675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:46.953396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34985 (local address 127.0.0.1:35835)
0620 00:10:46.953733 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:46.953744 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:46.954034 (+ 290us) server_negotiation.cc:407] Connection header received
0620 00:10:46.954181 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:46.954189 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:46.954270 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:46.954419 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:46.955128 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:46.956443 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:46.957596 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:46.958206 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:46.958305 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:46.961212 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:46.961216 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:46.961216 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:10:46.961230 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:46.963346 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:46.963836 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:46.963839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:46.963840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:46.963949 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:46.964208 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:46.964209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:46.964209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:46.964510 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:46.964719 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:46.964873 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 00:10:46.964977 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":84,"threads_started":1}
I20260620 00:10:46.966274 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:10:46.972435 20670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2195e0b5078245daa2ffa1a4df4ebdba: Generated new TSK 0
I20260620 00:10:46.972644 20670 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:10:46.979089 20675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:46.967704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40651 (local address 127.0.0.1:35835)
0620 00:10:46.967740 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:46.967741 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:46.972160 (+ 4419us) server_negotiation.cc:407] Connection header received
0620 00:10:46.972232 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:46.972233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:46.972243 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:46.972298 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:46.973005 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:46.973857 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:46.974623 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:46.974841 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:46.974918 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:46.976296 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:46.976297 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:46.976297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:10:46.976305 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:46.977951 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:46.978321 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:46.978322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:46.978322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:46.978385 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:46.978585 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:46.978586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:46.978586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:46.978718 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:46.978818 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:46.978920 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 00:10:46.978989 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 00:10:46.981168 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 20594
W20260620 00:10:46.985849 20579 connection.cc:535] client connection to 127.0.0.1:35835 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35835): Connection reset by peer (error 104)
2026-06-20T00:10:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1324 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:10:47 dist-test-slave-grqt krb5kdc[20693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:10:47 dist-test-slave-grqt krb5kdc[20693](info): set up 2 sockets
Jun 20 00:10:47 dist-test-slave-grqt krb5kdc[20693](info): commencing operation
krb5kdc: starting...
W20260620 00:10:48.009563 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:10:48 dist-test-slave-grqt krb5kdc[20693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:10:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:10:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 00:10:48.130389 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41747
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41747
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:10:48.215934 20706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:10:48.216162 20706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:10:48.216187 20706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:10:48.217437 20706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:10:48.217489 20706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:10:48.217504 20706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:10:48.217516 20706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:10:48.217528 20706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:10:48.219111 20706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41747
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:10:48.219393 20706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:10:48.219656 20706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:10:48.224709 20712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:48.224753 20711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:48.224776 20706 server_base.cc:1034] running on GCE node
W20260620 00:10:48.224951 20714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:48.225236 20706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:10:48.225579 20706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:10:48.226773 20706 hybrid_clock.cc:648] HybridClock initialized: now 1781914248226764 us; error 35 us; skew 500 ppm
Jun 20 00:10:48 dist-test-slave-grqt krb5kdc[20693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:10:48.236088 20706 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 00:10:48.236778 20706 webserver.cc:460] Webserver started at http://127.0.0.1:42403/ using document root <none> and password file <none>
I20260620 00:10:48.237398 20706 fs_manager.cc:359] Metadata directory not provided
I20260620 00:10:48.237525 20706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:10:48.237716 20706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:10:48.239511 20706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "2602026a0cfd491d9ebd46c3e76c246a"
format_stamp: "Formatted at 2026-06-20 00:10:48 on dist-test-slave-grqt"
server_key: "6e87c7a80a54139bb8feb3b4ed056af9"
server_key_iv: "829fcc50377dc0e758d6fa402bd90b84"
server_key_version: "encryptionkey@0"
I20260620 00:10:48.240211 20706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "2602026a0cfd491d9ebd46c3e76c246a"
format_stamp: "Formatted at 2026-06-20 00:10:48 on dist-test-slave-grqt"
server_key: "6e87c7a80a54139bb8feb3b4ed056af9"
server_key_iv: "829fcc50377dc0e758d6fa402bd90b84"
server_key_version: "encryptionkey@0"
I20260620 00:10:48.242332 20706 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 00:10:48.243960 20721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:48.244171 20706 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:10:48.244272 20706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "2602026a0cfd491d9ebd46c3e76c246a"
format_stamp: "Formatted at 2026-06-20 00:10:48 on dist-test-slave-grqt"
server_key: "6e87c7a80a54139bb8feb3b4ed056af9"
server_key_iv: "829fcc50377dc0e758d6fa402bd90b84"
server_key_version: "encryptionkey@0"
I20260620 00:10:48.244357 20706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:10:48.256740 20706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:10:48.259613 20706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:10:48.259789 20706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:10:48.265424 20706 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41747
I20260620 00:10:48.265565 20773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41747 every 8 connection(s)
I20260620 00:10:48.265936 20706 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:10:48.267095 20774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:10:48.270905 20774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2602026a0cfd491d9ebd46c3e76c246a: Bootstrap starting.
I20260620 00:10:48.271731 20774 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2602026a0cfd491d9ebd46c3e76c246a: Neither blocks nor log segments found. Creating new log.
I20260620 00:10:48.272097 20774 log.cc:824] T 00000000000000000000000000000000 P 2602026a0cfd491d9ebd46c3e76c246a: Log is configured to *not* fsync() on all Append() calls
I20260620 00:10:48.272997 20774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2602026a0cfd491d9ebd46c3e76c246a: No bootstrap required, opened a new log
I20260620 00:10:48.275398 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 20706
I20260620 00:10:48.275532 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:10:48.275781 392 external_mini_cluster.cc:1428] Setting key 44aded82207e39b192d4999ec72f40d3
I20260620 00:10:48.275959 20774 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2602026a0cfd491d9ebd46c3e76c246a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2602026a0cfd491d9ebd46c3e76c246a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41747 } }
I20260620 00:10:48.276168 20774 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2602026a0cfd491d9ebd46c3e76c246a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:10:48.276475 20774 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2602026a0cfd491d9ebd46c3e76c246a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2602026a0cfd491d9ebd46c3e76c246a, State: Initialized, Role: FOLLOWER
I20260620 00:10:48.276646 20774 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2602026a0cfd491d9ebd46c3e76c246a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2602026a0cfd491d9ebd46c3e76c246a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41747 } }
I20260620 00:10:48.276753 20774 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2602026a0cfd491d9ebd46c3e76c246a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:10:48.276796 20774 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2602026a0cfd491d9ebd46c3e76c246a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:10:48.276845 20774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2602026a0cfd491d9ebd46c3e76c246a [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:10:48.278256 20774 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2602026a0cfd491d9ebd46c3e76c246a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2602026a0cfd491d9ebd46c3e76c246a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41747 } }
I20260620 00:10:48.278384 20774 leader_election.cc:302] T 00000000000000000000000000000000 P 2602026a0cfd491d9ebd46c3e76c246a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2602026a0cfd491d9ebd46c3e76c246a; no voters:
I20260620 00:10:48.278577 20774 leader_election.cc:288] T 00000000000000000000000000000000 P 2602026a0cfd491d9ebd46c3e76c246a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:10:48.278602 20779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2602026a0cfd491d9ebd46c3e76c246a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:10:48.278708 20779 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2602026a0cfd491d9ebd46c3e76c246a [term 1 LEADER]: Becoming Leader. State: Replica: 2602026a0cfd491d9ebd46c3e76c246a, State: Running, Role: LEADER
I20260620 00:10:48.278791 20779 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2602026a0cfd491d9ebd46c3e76c246a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2602026a0cfd491d9ebd46c3e76c246a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41747 } }
I20260620 00:10:48.278841 20774 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2602026a0cfd491d9ebd46c3e76c246a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:10:48.279110 20779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2602026a0cfd491d9ebd46c3e76c246a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2602026a0cfd491d9ebd46c3e76c246a. Latest consensus state: current_term: 1 leader_uuid: "2602026a0cfd491d9ebd46c3e76c246a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2602026a0cfd491d9ebd46c3e76c246a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41747 } } }
I20260620 00:10:48.279181 20779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2602026a0cfd491d9ebd46c3e76c246a [sys.catalog]: This master's current role is: LEADER
I20260620 00:10:48.279847 20784 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:10:48.280342 20784 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:10:48.281450 20781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2602026a0cfd491d9ebd46c3e76c246a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2602026a0cfd491d9ebd46c3e76c246a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2602026a0cfd491d9ebd46c3e76c246a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41747 } } }
I20260620 00:10:48.281991 20781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2602026a0cfd491d9ebd46c3e76c246a [sys.catalog]: This master's current role is: LEADER
I20260620 00:10:48.282119 20784 catalog_manager.cc:1347] Generated new cluster ID: 1e2410d5868c49a5b20b9d501894b7f6
I20260620 00:10:48.282161 20784 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:10:48.282706 20778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:48.276299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51005 (local address 127.0.0.1:41747)
0620 00:10:48.276509 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:48.276520 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:48.276556 (+ 36us) server_negotiation.cc:407] Connection header received
0620 00:10:48.276693 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:48.276699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:48.276760 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:48.276903 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:48.277222 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:48.278737 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:48.280063 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:48.280669 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:48.280755 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:48.281926 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:48.281930 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:48.281930 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:10:48.282179 (+ 249us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:48.282222 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:10:48.282238 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:48.282331 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:48.282464 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 00:10:48.282495 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":97,"threads_started":1}
I20260620 00:10:48.283687 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:10:48.288627 20778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:48.285104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47479 (local address 127.0.0.1:41747)
0620 00:10:48.285143 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:48.285145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:48.285265 (+ 120us) server_negotiation.cc:407] Connection header received
0620 00:10:48.285295 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:48.285296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:48.285307 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:48.285406 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:48.285524 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:48.286258 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:48.287245 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:48.287574 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:48.287652 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:48.288358 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:48.288359 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:48.288360 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:10:48.288364 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:48.288380 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:10:48.288385 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:48.288444 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:48.288533 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 00:10:48.288547 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 00:10:48.306674 20784 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:10:48.307469 20784 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:10:48.311198 20784 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2602026a0cfd491d9ebd46c3e76c246a: Generated new TSK 0
I20260620 00:10:48.311419 20784 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:10:48.322574 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 20706
W20260620 00:10:48.326891 20691 connection.cc:535] client connection to 127.0.0.1:41747 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41747): Connection reset by peer (error 104)
2026-06-20T00:10:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1340 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:10:48 dist-test-slave-grqt krb5kdc[20805](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:10:48 dist-test-slave-grqt krb5kdc[20805](info): set up 2 sockets
Jun 20 00:10:48 dist-test-slave-grqt krb5kdc[20805](info): commencing operation
krb5kdc: starting...
W20260620 00:10:49.357462 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:10:49 dist-test-slave-grqt krb5kdc[20805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:10:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:10:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 00:10:49.476351 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46235
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:10:49.557781 20818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:10:49.557991 20818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:10:49.558013 20818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:10:49.559170 20818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:10:49.559213 20818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:10:49.559228 20818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:10:49.559267 20818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:10:49.559294 20818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:10:49.560720 20818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46235
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:10:49.560978 20818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:10:49.561206 20818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:10:49.565979 20824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:49.565966 20823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:49.565966 20826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:49.566236 20818 server_base.cc:1034] running on GCE node
I20260620 00:10:49.566449 20818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:10:49.566689 20818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:10:49.567857 20818 hybrid_clock.cc:648] HybridClock initialized: now 1781914249567843 us; error 30 us; skew 500 ppm
Jun 20 00:10:49 dist-test-slave-grqt krb5kdc[20805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:10:49.570282 20818 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 00:10:49.570760 20818 webserver.cc:460] Webserver started at http://127.0.0.1:43301/ using document root <none> and password file <none>
I20260620 00:10:49.571002 20818 fs_manager.cc:359] Metadata directory not provided
I20260620 00:10:49.571074 20818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:10:49.571236 20818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:10:49.572289 20818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "d3159554c1d948728f4ab141220ce214"
format_stamp: "Formatted at 2026-06-20 00:10:49 on dist-test-slave-grqt"
server_key: "e7dfbad563cb183e84fc2e3c3ea42f10"
server_key_iv: "efeecef91846f5dac352e7cd6b5f0c78"
server_key_version: "encryptionkey@0"
I20260620 00:10:49.572654 20818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "d3159554c1d948728f4ab141220ce214"
format_stamp: "Formatted at 2026-06-20 00:10:49 on dist-test-slave-grqt"
server_key: "e7dfbad563cb183e84fc2e3c3ea42f10"
server_key_iv: "efeecef91846f5dac352e7cd6b5f0c78"
server_key_version: "encryptionkey@0"
I20260620 00:10:49.574136 20818 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:10:49.574962 20833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:49.575188 20818 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:10:49.575299 20818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "d3159554c1d948728f4ab141220ce214"
format_stamp: "Formatted at 2026-06-20 00:10:49 on dist-test-slave-grqt"
server_key: "e7dfbad563cb183e84fc2e3c3ea42f10"
server_key_iv: "efeecef91846f5dac352e7cd6b5f0c78"
server_key_version: "encryptionkey@0"
I20260620 00:10:49.575376 20818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:10:49.578744 20818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:10:49.580634 20818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:10:49.580785 20818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:10:49.584928 20818 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46235
I20260620 00:10:49.585000 20885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46235 every 8 connection(s)
I20260620 00:10:49.585321 20818 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:10:49.586272 20886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:10:49.588470 20886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3159554c1d948728f4ab141220ce214: Bootstrap starting.
I20260620 00:10:49.588989 20886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3159554c1d948728f4ab141220ce214: Neither blocks nor log segments found. Creating new log.
I20260620 00:10:49.589223 20886 log.cc:824] T 00000000000000000000000000000000 P d3159554c1d948728f4ab141220ce214: Log is configured to *not* fsync() on all Append() calls
I20260620 00:10:49.589905 20886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3159554c1d948728f4ab141220ce214: No bootstrap required, opened a new log
I20260620 00:10:49.590981 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 20818
I20260620 00:10:49.591106 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:10:49.591300 392 external_mini_cluster.cc:1428] Setting key cdf590ff49e13214aed60416148e053a
I20260620 00:10:49.591998 20886 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3159554c1d948728f4ab141220ce214 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3159554c1d948728f4ab141220ce214" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46235 } }
I20260620 00:10:49.592149 20886 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3159554c1d948728f4ab141220ce214 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:10:49.592173 20886 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3159554c1d948728f4ab141220ce214 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3159554c1d948728f4ab141220ce214, State: Initialized, Role: FOLLOWER
I20260620 00:10:49.592315 20886 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3159554c1d948728f4ab141220ce214 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3159554c1d948728f4ab141220ce214" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46235 } }
I20260620 00:10:49.592412 20886 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3159554c1d948728f4ab141220ce214 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:10:49.592468 20886 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3159554c1d948728f4ab141220ce214 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:10:49.592521 20886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3159554c1d948728f4ab141220ce214 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:10:49.593539 20890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:49.591847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39731 (local address 127.0.0.1:46235)
0620 00:10:49.592028 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:49.592036 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:49.592068 (+ 32us) server_negotiation.cc:407] Connection header received
0620 00:10:49.592185 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:49.592193 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:49.592223 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:49.592389 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:49.593051 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:49.593055 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:49.593055 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:10:49.593059 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:49.593100 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:10:49.593111 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:49.593164 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:49.593243 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 00:10:49.593297 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":92,"threads_started":1}
I20260620 00:10:49.593677 20886 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3159554c1d948728f4ab141220ce214 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3159554c1d948728f4ab141220ce214" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46235 } }
I20260620 00:10:49.593788 20886 leader_election.cc:302] T 00000000000000000000000000000000 P d3159554c1d948728f4ab141220ce214 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3159554c1d948728f4ab141220ce214; no voters:
I20260620 00:10:49.593942 20886 leader_election.cc:288] T 00000000000000000000000000000000 P d3159554c1d948728f4ab141220ce214 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:10:49.593995 20891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3159554c1d948728f4ab141220ce214 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:10:49.594129 20891 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3159554c1d948728f4ab141220ce214 [term 1 LEADER]: Becoming Leader. State: Replica: d3159554c1d948728f4ab141220ce214, State: Running, Role: LEADER
I20260620 00:10:49.594201 20886 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3159554c1d948728f4ab141220ce214 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:10:49.594211 20891 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3159554c1d948728f4ab141220ce214 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3159554c1d948728f4ab141220ce214" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46235 } }
I20260620 00:10:49.594537 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:10:49.594565 20892 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3159554c1d948728f4ab141220ce214 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3159554c1d948728f4ab141220ce214" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3159554c1d948728f4ab141220ce214" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46235 } } }
I20260620 00:10:49.594635 20892 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3159554c1d948728f4ab141220ce214 [sys.catalog]: This master's current role is: LEADER
I20260620 00:10:49.594847 20893 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3159554c1d948728f4ab141220ce214 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3159554c1d948728f4ab141220ce214. Latest consensus state: current_term: 1 leader_uuid: "d3159554c1d948728f4ab141220ce214" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3159554c1d948728f4ab141220ce214" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46235 } } }
I20260620 00:10:49.595249 20893 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3159554c1d948728f4ab141220ce214 [sys.catalog]: This master's current role is: LEADER
I20260620 00:10:49.595480 20896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:10:49.596112 20896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:10:49.597184 20890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:49.596107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38381 (local address 127.0.0.1:46235)
0620 00:10:49.596144 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:49.596146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:49.596245 (+ 99us) server_negotiation.cc:407] Connection header received
0620 00:10:49.596304 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:49.596305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:49.596310 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:49.596370 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:49.596964 (+ 594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:49.596965 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:49.596965 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:10:49.596969 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:49.596984 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:10:49.596988 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:49.597053 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:49.597106 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 00:10:49.597120 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 00:10:49.597949 20896 catalog_manager.cc:1347] Generated new cluster ID: d739540a70154a4ea2c1a26684b86d22
I20260620 00:10:49.598006 20896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:10:49.603111 20896 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:10:49.603264 20896 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:10:49.609385 20896 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3159554c1d948728f4ab141220ce214: Generated new TSK 0
I20260620 00:10:49.609570 20896 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:10:49.614732 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 20818
2026-06-20T00:10:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1291 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:10:49 dist-test-slave-grqt krb5kdc[20917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:10:49 dist-test-slave-grqt krb5kdc[20917](info): set up 2 sockets
Jun 20 00:10:49 dist-test-slave-grqt krb5kdc[20917](info): commencing operation
krb5kdc: starting...
W20260620 00:10:50.645848 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:10:50 dist-test-slave-grqt krb5kdc[20917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:10:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:10:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 00:10:50.764384 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41325
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:10:50.851120 20930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:10:50.851336 20930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:10:50.851358 20930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:10:50.852456 20930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:10:50.852495 20930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:10:50.852510 20930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:10:50.852522 20930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:10:50.852535 20930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:10:50.854056 20930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:10:50.854264 20930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:10:50.854434 20930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:10:50.859393 20935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:50.859416 20938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:50.859418 20936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:50.859678 20930 server_base.cc:1034] running on GCE node
I20260620 00:10:50.859941 20930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:10:50.860189 20930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:10:50.861337 20930 hybrid_clock.cc:648] HybridClock initialized: now 1781914250861307 us; error 49 us; skew 500 ppm
Jun 20 00:10:50 dist-test-slave-grqt krb5kdc[20917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:10:50.864027 20930 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 00:10:50.864549 20930 webserver.cc:460] Webserver started at http://127.0.0.1:36441/ using document root <none> and password file <none>
I20260620 00:10:50.864792 20930 fs_manager.cc:359] Metadata directory not provided
I20260620 00:10:50.864836 20930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:10:50.864997 20930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:10:50.866130 20930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "4dafb8122c4949138d8def2bbdbf9d5b"
format_stamp: "Formatted at 2026-06-20 00:10:50 on dist-test-slave-grqt"
server_key: "75742e40be7cb1c83bc91505f8d9ed82"
server_key_iv: "8f33c585c3931e26652639a38de96f09"
server_key_version: "encryptionkey@0"
I20260620 00:10:50.866503 20930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "4dafb8122c4949138d8def2bbdbf9d5b"
format_stamp: "Formatted at 2026-06-20 00:10:50 on dist-test-slave-grqt"
server_key: "75742e40be7cb1c83bc91505f8d9ed82"
server_key_iv: "8f33c585c3931e26652639a38de96f09"
server_key_version: "encryptionkey@0"
I20260620 00:10:50.867842 20930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:10:50.868685 20945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:50.868914 20930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:10:50.869012 20930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "4dafb8122c4949138d8def2bbdbf9d5b"
format_stamp: "Formatted at 2026-06-20 00:10:50 on dist-test-slave-grqt"
server_key: "75742e40be7cb1c83bc91505f8d9ed82"
server_key_iv: "8f33c585c3931e26652639a38de96f09"
server_key_version: "encryptionkey@0"
I20260620 00:10:50.869117 20930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:10:50.892613 20930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:10:50.894987 20930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:10:50.895149 20930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:10:50.899343 20930 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41325
I20260620 00:10:50.899412 20997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41325 every 8 connection(s)
I20260620 00:10:50.899745 20930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:10:50.900609 20998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:10:50.902947 20998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dafb8122c4949138d8def2bbdbf9d5b: Bootstrap starting.
I20260620 00:10:50.903524 20998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4dafb8122c4949138d8def2bbdbf9d5b: Neither blocks nor log segments found. Creating new log.
I20260620 00:10:50.903816 20998 log.cc:824] T 00000000000000000000000000000000 P 4dafb8122c4949138d8def2bbdbf9d5b: Log is configured to *not* fsync() on all Append() calls
I20260620 00:10:50.904565 20998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dafb8122c4949138d8def2bbdbf9d5b: No bootstrap required, opened a new log
I20260620 00:10:50.906303 20998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4dafb8122c4949138d8def2bbdbf9d5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dafb8122c4949138d8def2bbdbf9d5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41325 } }
I20260620 00:10:50.906457 20998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4dafb8122c4949138d8def2bbdbf9d5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:10:50.906481 20998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4dafb8122c4949138d8def2bbdbf9d5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dafb8122c4949138d8def2bbdbf9d5b, State: Initialized, Role: FOLLOWER
I20260620 00:10:50.906636 20998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4dafb8122c4949138d8def2bbdbf9d5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dafb8122c4949138d8def2bbdbf9d5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41325 } }
I20260620 00:10:50.906713 20998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4dafb8122c4949138d8def2bbdbf9d5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:10:50.906735 20998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4dafb8122c4949138d8def2bbdbf9d5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:10:50.906787 20998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4dafb8122c4949138d8def2bbdbf9d5b [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:10:50.907352 20998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4dafb8122c4949138d8def2bbdbf9d5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dafb8122c4949138d8def2bbdbf9d5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41325 } }
I20260620 00:10:50.907444 20998 leader_election.cc:302] T 00000000000000000000000000000000 P 4dafb8122c4949138d8def2bbdbf9d5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4dafb8122c4949138d8def2bbdbf9d5b; no voters:
I20260620 00:10:50.907657 20998 leader_election.cc:288] T 00000000000000000000000000000000 P 4dafb8122c4949138d8def2bbdbf9d5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:10:50.907707 21001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4dafb8122c4949138d8def2bbdbf9d5b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:10:50.907909 21001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4dafb8122c4949138d8def2bbdbf9d5b [term 1 LEADER]: Becoming Leader. State: Replica: 4dafb8122c4949138d8def2bbdbf9d5b, State: Running, Role: LEADER
I20260620 00:10:50.907951 20998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4dafb8122c4949138d8def2bbdbf9d5b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:10:50.908004 21001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4dafb8122c4949138d8def2bbdbf9d5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dafb8122c4949138d8def2bbdbf9d5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41325 } }
I20260620 00:10:50.908473 21002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dafb8122c4949138d8def2bbdbf9d5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4dafb8122c4949138d8def2bbdbf9d5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dafb8122c4949138d8def2bbdbf9d5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41325 } } }
I20260620 00:10:50.908511 21003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dafb8122c4949138d8def2bbdbf9d5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4dafb8122c4949138d8def2bbdbf9d5b. Latest consensus state: current_term: 1 leader_uuid: "4dafb8122c4949138d8def2bbdbf9d5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dafb8122c4949138d8def2bbdbf9d5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41325 } } }
I20260620 00:10:50.908603 21002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dafb8122c4949138d8def2bbdbf9d5b [sys.catalog]: This master's current role is: LEADER
I20260620 00:10:50.908614 21003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dafb8122c4949138d8def2bbdbf9d5b [sys.catalog]: This master's current role is: LEADER
I20260620 00:10:50.908840 21005 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:10:50.909438 21005 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:10:50.909767 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 20930
I20260620 00:10:50.909931 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:10:50.910120 392 external_mini_cluster.cc:1428] Setting key 5f5e046a94569be211e33f2fd2f3c7a8
I20260620 00:10:50.912180 21005 catalog_manager.cc:1347] Generated new cluster ID: 7abff0ef6a394eb7bf8c667157650c4f
I20260620 00:10:50.912236 21005 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 00:10:50 dist-test-slave-grqt krb5kdc[20917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 00:10:50.920454 21020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:50.910775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41621 (local address 127.0.0.1:41325)
0620 00:10:50.910968 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:50.910977 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:50.911006 (+ 29us) server_negotiation.cc:407] Connection header received
0620 00:10:50.911154 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:50.911161 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:50.911216 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:50.912158 (+ 942us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:50.912256 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:50.913208 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:50.914047 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:50.914493 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:50.914630 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:50.917009 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:50.917014 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:50.917015 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:10:50.917028 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:50.918834 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:50.919226 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:50.919229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:50.919229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:50.919339 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:50.919555 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:50.919556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:50.919556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:50.919820 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:50.919999 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:50.920072 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 00:10:50.920244 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":66,"threads_started":1}
I20260620 00:10:50.921433 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:10:50.928959 21020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:50.922779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42439 (local address 127.0.0.1:41325)
0620 00:10:50.922814 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:50.922816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:50.922995 (+ 179us) server_negotiation.cc:407] Connection header received
0620 00:10:50.923066 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:50.923067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:50.923079 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:50.923134 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:50.923654 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:50.924163 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:50.924866 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:50.925093 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:50.925179 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:50.926408 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:50.926409 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:50.926409 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:10:50.926416 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:50.927817 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:50.928172 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:50.928173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:50.928174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:50.928228 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:50.928432 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:50.928433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:50.928433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:50.928546 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:50.928675 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:50.928773 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 00:10:50.928849 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 00:10:50.937261 21005 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:10:50.937824 21005 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:10:50.941445 21005 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4dafb8122c4949138d8def2bbdbf9d5b: Generated new TSK 0
I20260620 00:10:50.941634 21005 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:10:50.950081 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 20930
2026-06-20T00:10:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1335 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:10:51 dist-test-slave-grqt krb5kdc[21029](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:10:51 dist-test-slave-grqt krb5kdc[21029](info): set up 2 sockets
Jun 20 00:10:51 dist-test-slave-grqt krb5kdc[21029](info): commencing operation
krb5kdc: starting...
W20260620 00:10:51.989667 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:10:52 dist-test-slave-grqt krb5kdc[21029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:10:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:10:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 00:10:52.109872 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45383
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:10:52.198489 21042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:10:52.198704 21042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:10:52.198729 21042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:10:52.199833 21042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:10:52.199875 21042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:10:52.199889 21042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:10:52.199903 21042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:10:52.199914 21042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:10:52.201452 21042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:10:52.201663 21042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:10:52.201843 21042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:10:52.206635 21048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:52.206653 21047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:52.206635 21050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:52.206907 21042 server_base.cc:1034] running on GCE node
I20260620 00:10:52.207144 21042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:10:52.207430 21042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:10:52.208575 21042 hybrid_clock.cc:648] HybridClock initialized: now 1781914252208559 us; error 32 us; skew 500 ppm
Jun 20 00:10:52 dist-test-slave-grqt krb5kdc[21029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:10:52.211127 21042 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 00:10:52.211570 21042 webserver.cc:460] Webserver started at http://127.0.0.1:40129/ using document root <none> and password file <none>
I20260620 00:10:52.211833 21042 fs_manager.cc:359] Metadata directory not provided
I20260620 00:10:52.211905 21042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:10:52.212055 21042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:10:52.213126 21042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "3c977d32490942b1abc92c2a69fa69e9"
format_stamp: "Formatted at 2026-06-20 00:10:52 on dist-test-slave-grqt"
server_key: "461d5ac50fd238858aff37bcdc1cbc85"
server_key_iv: "160ba346ac6263ed969a73dc1b442a64"
server_key_version: "encryptionkey@0"
I20260620 00:10:52.213533 21042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "3c977d32490942b1abc92c2a69fa69e9"
format_stamp: "Formatted at 2026-06-20 00:10:52 on dist-test-slave-grqt"
server_key: "461d5ac50fd238858aff37bcdc1cbc85"
server_key_iv: "160ba346ac6263ed969a73dc1b442a64"
server_key_version: "encryptionkey@0"
I20260620 00:10:52.214898 21042 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:10:52.215623 21057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:52.215793 21042 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 00:10:52.215901 21042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "3c977d32490942b1abc92c2a69fa69e9"
format_stamp: "Formatted at 2026-06-20 00:10:52 on dist-test-slave-grqt"
server_key: "461d5ac50fd238858aff37bcdc1cbc85"
server_key_iv: "160ba346ac6263ed969a73dc1b442a64"
server_key_version: "encryptionkey@0"
I20260620 00:10:52.215978 21042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:10:52.233516 21042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:10:52.235601 21042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:10:52.235781 21042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:10:52.239794 21042 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45383
I20260620 00:10:52.239833 21109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45383 every 8 connection(s)
I20260620 00:10:52.240211 21042 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:10:52.241190 21110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:10:52.243854 21110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c977d32490942b1abc92c2a69fa69e9: Bootstrap starting.
I20260620 00:10:52.244491 21110 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c977d32490942b1abc92c2a69fa69e9: Neither blocks nor log segments found. Creating new log.
I20260620 00:10:52.244788 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 21042
I20260620 00:10:52.244827 21110 log.cc:824] T 00000000000000000000000000000000 P 3c977d32490942b1abc92c2a69fa69e9: Log is configured to *not* fsync() on all Append() calls
I20260620 00:10:52.244928 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:10:52.245193 392 external_mini_cluster.cc:1428] Setting key 6c3770ef25f812afa0d51d96f63696af
I20260620 00:10:52.245771 21110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c977d32490942b1abc92c2a69fa69e9: No bootstrap required, opened a new log
I20260620 00:10:52.248862 21110 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c977d32490942b1abc92c2a69fa69e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c977d32490942b1abc92c2a69fa69e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45383 } }
I20260620 00:10:52.249058 21110 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c977d32490942b1abc92c2a69fa69e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:10:52.249091 21110 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c977d32490942b1abc92c2a69fa69e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c977d32490942b1abc92c2a69fa69e9, State: Initialized, Role: FOLLOWER
I20260620 00:10:52.249229 21110 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c977d32490942b1abc92c2a69fa69e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c977d32490942b1abc92c2a69fa69e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45383 } }
I20260620 00:10:52.249328 21110 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c977d32490942b1abc92c2a69fa69e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:10:52.249421 21110 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c977d32490942b1abc92c2a69fa69e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:10:52.249480 21110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c977d32490942b1abc92c2a69fa69e9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:10:52.250114 21110 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c977d32490942b1abc92c2a69fa69e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c977d32490942b1abc92c2a69fa69e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45383 } }
I20260620 00:10:52.250226 21110 leader_election.cc:302] T 00000000000000000000000000000000 P 3c977d32490942b1abc92c2a69fa69e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c977d32490942b1abc92c2a69fa69e9; no voters:
I20260620 00:10:52.250389 21110 leader_election.cc:288] T 00000000000000000000000000000000 P 3c977d32490942b1abc92c2a69fa69e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:10:52.250453 21115 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c977d32490942b1abc92c2a69fa69e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:10:52.250622 21115 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c977d32490942b1abc92c2a69fa69e9 [term 1 LEADER]: Becoming Leader. State: Replica: 3c977d32490942b1abc92c2a69fa69e9, State: Running, Role: LEADER
I20260620 00:10:52.250640 21110 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c977d32490942b1abc92c2a69fa69e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:10:52.250708 21115 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c977d32490942b1abc92c2a69fa69e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c977d32490942b1abc92c2a69fa69e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45383 } }
I20260620 00:10:52.251006 21115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c977d32490942b1abc92c2a69fa69e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c977d32490942b1abc92c2a69fa69e9. Latest consensus state: current_term: 1 leader_uuid: "3c977d32490942b1abc92c2a69fa69e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c977d32490942b1abc92c2a69fa69e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45383 } } }
I20260620 00:10:52.251132 21115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c977d32490942b1abc92c2a69fa69e9 [sys.catalog]: This master's current role is: LEADER
I20260620 00:10:52.251340 21121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:10:52.251317 21116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c977d32490942b1abc92c2a69fa69e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c977d32490942b1abc92c2a69fa69e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c977d32490942b1abc92c2a69fa69e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45383 } } }
I20260620 00:10:52.251405 21116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c977d32490942b1abc92c2a69fa69e9 [sys.catalog]: This master's current role is: LEADER
I20260620 00:10:52.251878 21121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:10:52.252687 21114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:52.245796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60717 (local address 127.0.0.1:45383)
0620 00:10:52.245977 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:52.245987 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:52.246026 (+ 39us) server_negotiation.cc:407] Connection header received
0620 00:10:52.247159 (+ 1133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:52.247169 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:52.247246 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:52.247417 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:52.247599 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:52.249201 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:52.250129 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:52.250874 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:52.250995 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:52.251799 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:52.251804 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:52.251804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:10:52.252045 (+ 241us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:52.252086 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:10:52.252103 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:52.252181 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:52.252384 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 00:10:52.252454 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":85,"threads_started":1}
I20260620 00:10:52.253319 21131 mvcc.cc:205] Tried to move back new op lower bound from 7298720777227616256 to 7298720777219248128. Current Snapshot: MvccSnapshot[applied={T|T < 7298720777227616256}]
I20260620 00:10:52.253708 21121 catalog_manager.cc:1347] Generated new cluster ID: 2c94e60e6c06479f953ef889e902c8ac
I20260620 00:10:52.253757 21121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:10:52.253902 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:10:52.259043 21114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:52.255513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45549 (local address 127.0.0.1:45383)
0620 00:10:52.255553 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:52.255555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:52.255591 (+ 36us) server_negotiation.cc:407] Connection header received
0620 00:10:52.255649 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:52.255650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:52.255660 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:52.255736 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:52.255836 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:52.256641 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:52.257621 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:52.257963 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:52.258048 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:52.258786 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:52.258787 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:52.258787 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:10:52.258792 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:52.258808 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:10:52.258813 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:52.258880 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:52.258939 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 00:10:52.258961 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 00:10:52.263353 21121 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:10:52.264155 21121 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:10:52.273409 21121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c977d32490942b1abc92c2a69fa69e9: Generated new TSK 0
I20260620 00:10:52.273641 21121 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:10:52.275633 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 21042
2026-06-20T00:10:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:10:52 dist-test-slave-grqt krb5kdc[21141](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:10:52 dist-test-slave-grqt krb5kdc[21141](info): set up 2 sockets
Jun 20 00:10:52 dist-test-slave-grqt krb5kdc[21141](info): commencing operation
krb5kdc: starting...
W20260620 00:10:53.301746 392 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:10:53 dist-test-slave-grqt krb5kdc[21141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:10:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:10:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 00:10:53.418433 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40685
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:10:53.499394 21154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:10:53.499594 21154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:10:53.499616 21154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:10:53.500764 21154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:10:53.500805 21154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:10:53.500819 21154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:10:53.500873 21154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:10:53.500895 21154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:10:53.502413 21154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:10:53.502655 21154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:10:53.502913 21154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:10:53.507745 21160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:53.507871 21154 server_base.cc:1034] running on GCE node
W20260620 00:10:53.507737 21162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:53.507740 21159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:53.508260 21154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:10:53.508515 21154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:10:53.509660 21154 hybrid_clock.cc:648] HybridClock initialized: now 1781914253509649 us; error 34 us; skew 500 ppm
Jun 20 00:10:53 dist-test-slave-grqt krb5kdc[21141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:10:53.516460 21154 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 00:10:53.516884 21154 webserver.cc:460] Webserver started at http://127.0.0.1:36667/ using document root <none> and password file <none>
I20260620 00:10:53.517089 21154 fs_manager.cc:359] Metadata directory not provided
I20260620 00:10:53.517131 21154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:10:53.517232 21154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:10:53.518322 21154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "b2f2863ec160476caa41b24692b91288"
format_stamp: "Formatted at 2026-06-20 00:10:53 on dist-test-slave-grqt"
server_key: "d5f7ff0edeb503b0040b325ca6e47ab9"
server_key_iv: "1791a04e4eb6d929483587f4c30ead77"
server_key_version: "encryptionkey@0"
I20260620 00:10:53.518623 21154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "b2f2863ec160476caa41b24692b91288"
format_stamp: "Formatted at 2026-06-20 00:10:53 on dist-test-slave-grqt"
server_key: "d5f7ff0edeb503b0040b325ca6e47ab9"
server_key_iv: "1791a04e4eb6d929483587f4c30ead77"
server_key_version: "encryptionkey@0"
I20260620 00:10:53.519939 21154 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:10:53.520648 21169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:53.520841 21154 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:10:53.520951 21154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "b2f2863ec160476caa41b24692b91288"
format_stamp: "Formatted at 2026-06-20 00:10:53 on dist-test-slave-grqt"
server_key: "d5f7ff0edeb503b0040b325ca6e47ab9"
server_key_iv: "1791a04e4eb6d929483587f4c30ead77"
server_key_version: "encryptionkey@0"
I20260620 00:10:53.521027 21154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:10:53.524355 21154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:10:53.526336 21154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:10:53.526489 21154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:10:53.530467 21154 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40685
I20260620 00:10:53.530516 21221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40685 every 8 connection(s)
I20260620 00:10:53.530848 21154 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:10:53.531752 21222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:10:53.533448 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 21154
I20260620 00:10:53.533576 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:10:53.533780 392 external_mini_cluster.cc:1428] Setting key ffddd524f49f299a2e2118768cce5093
I20260620 00:10:53.534700 21222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2f2863ec160476caa41b24692b91288: Bootstrap starting.
I20260620 00:10:53.535372 21222 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2f2863ec160476caa41b24692b91288: Neither blocks nor log segments found. Creating new log.
I20260620 00:10:53.535712 21222 log.cc:824] T 00000000000000000000000000000000 P b2f2863ec160476caa41b24692b91288: Log is configured to *not* fsync() on all Append() calls
I20260620 00:10:53.536274 21225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:53.534347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59733 (local address 127.0.0.1:40685)
0620 00:10:53.534517 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:53.534524 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:53.534549 (+ 25us) server_negotiation.cc:407] Connection header received
0620 00:10:53.534988 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:53.534994 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:53.535020 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:53.535136 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:53.535844 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:53.535852 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:53.535852 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:10:53.535856 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:53.535890 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:10:53.535900 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:53.536006 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:53.536053 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 00:10:53.536094 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":83,"threads_started":1}
I20260620 00:10:53.536474 21222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2f2863ec160476caa41b24692b91288: No bootstrap required, opened a new log
I20260620 00:10:53.537199 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:10:53.539199 21222 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2f2863ec160476caa41b24692b91288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f2863ec160476caa41b24692b91288" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40685 } }
I20260620 00:10:53.539389 21222 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2f2863ec160476caa41b24692b91288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:10:53.539429 21222 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2f2863ec160476caa41b24692b91288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2f2863ec160476caa41b24692b91288, State: Initialized, Role: FOLLOWER
I20260620 00:10:53.539567 21222 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2f2863ec160476caa41b24692b91288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f2863ec160476caa41b24692b91288" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40685 } }
I20260620 00:10:53.539695 21222 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2f2863ec160476caa41b24692b91288 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:10:53.539736 21222 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2f2863ec160476caa41b24692b91288 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:10:53.539788 21222 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2f2863ec160476caa41b24692b91288 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:10:53.539893 21225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:53.538781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43697 (local address 127.0.0.1:40685)
0620 00:10:53.538825 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:53.538827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:53.538901 (+ 74us) server_negotiation.cc:407] Connection header received
0620 00:10:53.538970 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:53.538971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:53.538974 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:53.539014 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:53.539523 (+ 509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:53.539524 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:53.539524 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:10:53.539528 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:53.539556 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:10:53.539560 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:53.539611 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:53.539822 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 00:10:53.539835 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 00:10:53.540412 21222 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2f2863ec160476caa41b24692b91288 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f2863ec160476caa41b24692b91288" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40685 } }
I20260620 00:10:53.540545 21222 leader_election.cc:302] T 00000000000000000000000000000000 P b2f2863ec160476caa41b24692b91288 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2f2863ec160476caa41b24692b91288; no voters:
I20260620 00:10:53.540724 21222 leader_election.cc:288] T 00000000000000000000000000000000 P b2f2863ec160476caa41b24692b91288 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:10:53.540796 21232 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2f2863ec160476caa41b24692b91288 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:10:53.540937 21232 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2f2863ec160476caa41b24692b91288 [term 1 LEADER]: Becoming Leader. State: Replica: b2f2863ec160476caa41b24692b91288, State: Running, Role: LEADER
I20260620 00:10:53.540957 21222 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2f2863ec160476caa41b24692b91288 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:10:53.541024 21232 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2f2863ec160476caa41b24692b91288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f2863ec160476caa41b24692b91288" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40685 } }
I20260620 00:10:53.541433 21234 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2f2863ec160476caa41b24692b91288 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2f2863ec160476caa41b24692b91288. Latest consensus state: current_term: 1 leader_uuid: "b2f2863ec160476caa41b24692b91288" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f2863ec160476caa41b24692b91288" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40685 } } }
I20260620 00:10:53.541452 21233 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2f2863ec160476caa41b24692b91288 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2f2863ec160476caa41b24692b91288" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f2863ec160476caa41b24692b91288" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40685 } } }
I20260620 00:10:53.541570 21234 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2f2863ec160476caa41b24692b91288 [sys.catalog]: This master's current role is: LEADER
I20260620 00:10:53.541579 21233 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2f2863ec160476caa41b24692b91288 [sys.catalog]: This master's current role is: LEADER
I20260620 00:10:53.541798 21241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:10:53.542294 21241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:10:53.544056 21241 catalog_manager.cc:1347] Generated new cluster ID: a09dd2c671a44980b4b4f4a31eb2b9c5
I20260620 00:10:53.544108 21241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:10:53.555402 21241 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:10:53.555590 21241 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:10:53.559696 21241 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2f2863ec160476caa41b24692b91288: Generated new TSK 0
I20260620 00:10:53.559891 21241 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:10:53.565843 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 21154
2026-06-20T00:10:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1290 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:10:53 dist-test-slave-grqt krb5kdc[21253](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:10:53 dist-test-slave-grqt krb5kdc[21253](info): set up 2 sockets
Jun 20 00:10:53 dist-test-slave-grqt krb5kdc[21253](info): commencing operation
krb5kdc: starting...
W20260620 00:10:54.609601 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:10:54 dist-test-slave-grqt krb5kdc[21253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:10:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:10:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 00:10:54.730352 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35225
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:10:54.812683 21266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:10:54.812887 21266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:10:54.812909 21266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:10:54.814003 21266 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:10:54.814039 21266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:10:54.814054 21266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:10:54.814069 21266 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:10:54.814080 21266 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:10:54.815615 21266 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:10:54.815809 21266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:10:54.815972 21266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:10:54.820777 21274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:54.820907 21266 server_base.cc:1034] running on GCE node
W20260620 00:10:54.820782 21271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:54.820782 21272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:54.821242 21266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:10:54.821554 21266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:10:54.822724 21266 hybrid_clock.cc:648] HybridClock initialized: now 1781914254822718 us; error 36 us; skew 500 ppm
Jun 20 00:10:54 dist-test-slave-grqt krb5kdc[21253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:10:54.825291 21266 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 00:10:54.825793 21266 webserver.cc:460] Webserver started at http://127.0.0.1:33043/ using document root <none> and password file <none>
I20260620 00:10:54.826014 21266 fs_manager.cc:359] Metadata directory not provided
I20260620 00:10:54.826061 21266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:10:54.826229 21266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:10:54.827298 21266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "21b704c7e5434ee6ba8b116638ef764e"
format_stamp: "Formatted at 2026-06-20 00:10:54 on dist-test-slave-grqt"
server_key: "e04b636896834939ddf1009c951ce92a"
server_key_iv: "477a795413a6e607c8c17b20a9cf95fd"
server_key_version: "encryptionkey@0"
I20260620 00:10:54.827661 21266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "21b704c7e5434ee6ba8b116638ef764e"
format_stamp: "Formatted at 2026-06-20 00:10:54 on dist-test-slave-grqt"
server_key: "e04b636896834939ddf1009c951ce92a"
server_key_iv: "477a795413a6e607c8c17b20a9cf95fd"
server_key_version: "encryptionkey@0"
I20260620 00:10:54.828981 21266 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:10:54.829802 21281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:54.830175 21266 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:10:54.830294 21266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "21b704c7e5434ee6ba8b116638ef764e"
format_stamp: "Formatted at 2026-06-20 00:10:54 on dist-test-slave-grqt"
server_key: "e04b636896834939ddf1009c951ce92a"
server_key_iv: "477a795413a6e607c8c17b20a9cf95fd"
server_key_version: "encryptionkey@0"
I20260620 00:10:54.830374 21266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:10:54.841291 21266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:10:54.843853 21266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:10:54.844058 21266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:10:54.848656 21266 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35225
I20260620 00:10:54.848698 21333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35225 every 8 connection(s)
I20260620 00:10:54.849079 21266 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:10:54.850028 21334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:10:54.852399 21334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21b704c7e5434ee6ba8b116638ef764e: Bootstrap starting.
I20260620 00:10:54.852962 21334 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21b704c7e5434ee6ba8b116638ef764e: Neither blocks nor log segments found. Creating new log.
I20260620 00:10:54.853205 21334 log.cc:824] T 00000000000000000000000000000000 P 21b704c7e5434ee6ba8b116638ef764e: Log is configured to *not* fsync() on all Append() calls
I20260620 00:10:54.853981 21334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21b704c7e5434ee6ba8b116638ef764e: No bootstrap required, opened a new log
I20260620 00:10:54.855430 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 21266
I20260620 00:10:54.855571 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:10:54.855783 392 external_mini_cluster.cc:1428] Setting key ca614942bca96313f7db2ab6bf36c300
I20260620 00:10:54.855839 21334 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21b704c7e5434ee6ba8b116638ef764e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21b704c7e5434ee6ba8b116638ef764e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35225 } }
I20260620 00:10:54.856433 21334 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21b704c7e5434ee6ba8b116638ef764e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:10:54.856545 21334 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21b704c7e5434ee6ba8b116638ef764e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21b704c7e5434ee6ba8b116638ef764e, State: Initialized, Role: FOLLOWER
I20260620 00:10:54.856734 21334 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21b704c7e5434ee6ba8b116638ef764e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21b704c7e5434ee6ba8b116638ef764e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35225 } }
I20260620 00:10:54.856825 21334 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21b704c7e5434ee6ba8b116638ef764e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:10:54.856856 21334 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21b704c7e5434ee6ba8b116638ef764e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:10:54.856909 21334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21b704c7e5434ee6ba8b116638ef764e [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:10:54.858057 21334 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21b704c7e5434ee6ba8b116638ef764e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21b704c7e5434ee6ba8b116638ef764e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35225 } }
I20260620 00:10:54.858171 21334 leader_election.cc:302] T 00000000000000000000000000000000 P 21b704c7e5434ee6ba8b116638ef764e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21b704c7e5434ee6ba8b116638ef764e; no voters:
I20260620 00:10:54.858351 21334 leader_election.cc:288] T 00000000000000000000000000000000 P 21b704c7e5434ee6ba8b116638ef764e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:10:54.858428 21339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21b704c7e5434ee6ba8b116638ef764e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:10:54.858616 21334 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21b704c7e5434ee6ba8b116638ef764e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:10:54.858649 21339 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21b704c7e5434ee6ba8b116638ef764e [term 1 LEADER]: Becoming Leader. State: Replica: 21b704c7e5434ee6ba8b116638ef764e, State: Running, Role: LEADER
I20260620 00:10:54.858870 21339 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21b704c7e5434ee6ba8b116638ef764e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21b704c7e5434ee6ba8b116638ef764e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35225 } }
I20260620 00:10:54.859277 21340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21b704c7e5434ee6ba8b116638ef764e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21b704c7e5434ee6ba8b116638ef764e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21b704c7e5434ee6ba8b116638ef764e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35225 } } }
I20260620 00:10:54.859359 21340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21b704c7e5434ee6ba8b116638ef764e [sys.catalog]: This master's current role is: LEADER
I20260620 00:10:54.859522 21341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21b704c7e5434ee6ba8b116638ef764e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21b704c7e5434ee6ba8b116638ef764e. Latest consensus state: current_term: 1 leader_uuid: "21b704c7e5434ee6ba8b116638ef764e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21b704c7e5434ee6ba8b116638ef764e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35225 } } }
I20260620 00:10:54.859589 21341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21b704c7e5434ee6ba8b116638ef764e [sys.catalog]: This master's current role is: LEADER
I20260620 00:10:54.860014 21348 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:10:54.860723 21348 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 00:10:54 dist-test-slave-grqt krb5kdc[21253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 00:10:54.867390 21348 catalog_manager.cc:1347] Generated new cluster ID: c20620218cf24f1fbfc4944fc0baed10
I20260620 00:10:54.867466 21348 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:10:54.868959 21338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:54.856377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.57:56999 (local address 127.0.0.1:35225)
0620 00:10:54.856581 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:54.856591 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:54.856630 (+ 39us) server_negotiation.cc:407] Connection header received
0620 00:10:54.856757 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:54.856764 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:54.856826 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:54.856986 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:54.857887 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:54.859373 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:54.860585 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:54.861230 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:54.861345 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:54.864331 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:54.864336 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:54.864336 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:10:54.864350 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:54.866536 (+ 2186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:54.866981 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:54.866985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:54.866986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:54.867120 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:54.867429 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:54.867430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:54.867430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:54.867789 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:54.868491 (+ 702us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:54.868593 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 00:10:54.868725 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":110,"threads_started":1}
I20260620 00:10:54.870342 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:10:54.877664 21348 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:10:54.878417 21348 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:10:54.879516 21338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:54.871892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.57:35145 (local address 127.0.0.1:35225)
0620 00:10:54.871927 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:54.871929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:54.871938 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:10:54.871984 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:54.871985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:54.871994 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:54.872047 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:54.872668 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:54.873483 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:54.874465 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:54.874822 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:54.874897 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:54.876363 (+ 1466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:54.876365 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:54.876365 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:10:54.876372 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:54.878219 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:54.878681 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:54.878682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:54.878682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:54.878741 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:54.878978 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:54.878979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:54.878979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:54.879109 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:54.879204 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:54.879364 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 00:10:54.879427 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 00:10:54.885991 21348 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21b704c7e5434ee6ba8b116638ef764e: Generated new TSK 0
I20260620 00:10:54.886219 21348 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:10:54.893287 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 21266
W20260620 00:10:54.897506 21251 connection.cc:535] client connection to 127.0.0.1:35225 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35225): Connection reset by peer (error 104)
2026-06-20T00:10:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1327 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:10:54 dist-test-slave-grqt krb5kdc[21365](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:10:54 dist-test-slave-grqt krb5kdc[21365](info): set up 2 sockets
Jun 20 00:10:54 dist-test-slave-grqt krb5kdc[21365](info): commencing operation
krb5kdc: starting...
W20260620 00:10:55.949529 392 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:10:56 dist-test-slave-grqt krb5kdc[21365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:10:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:10:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 00:10:56.068027 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37247
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:10:56.149358 21378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:10:56.149595 21378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:10:56.149617 21378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:10:56.150692 21378 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:10:56.150727 21378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:10:56.150741 21378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:10:56.150754 21378 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:10:56.150766 21378 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:10:56.152140 21378 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:10:56.152323 21378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:10:56.152483 21378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:10:56.157392 21386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:56.157466 21378 server_base.cc:1034] running on GCE node
W20260620 00:10:56.157403 21384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:56.157418 21383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:56.157805 21378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:10:56.158102 21378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:10:56.159260 21378 hybrid_clock.cc:648] HybridClock initialized: now 1781914256159246 us; error 29 us; skew 500 ppm
Jun 20 00:10:56 dist-test-slave-grqt krb5kdc[21365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:10:56.161808 21378 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 00:10:56.162292 21378 webserver.cc:460] Webserver started at http://127.0.0.1:38557/ using document root <none> and password file <none>
I20260620 00:10:56.162518 21378 fs_manager.cc:359] Metadata directory not provided
I20260620 00:10:56.162585 21378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:10:56.162734 21378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:10:56.163777 21378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "bbe84cf52d9c4f7db150d727b2b95739"
format_stamp: "Formatted at 2026-06-20 00:10:56 on dist-test-slave-grqt"
server_key: "cb19998d71b0348d8b3d6bef24281726"
server_key_iv: "effe4d7b1faf99d7bada725ae2a16cff"
server_key_version: "encryptionkey@0"
I20260620 00:10:56.164166 21378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "bbe84cf52d9c4f7db150d727b2b95739"
format_stamp: "Formatted at 2026-06-20 00:10:56 on dist-test-slave-grqt"
server_key: "cb19998d71b0348d8b3d6bef24281726"
server_key_iv: "effe4d7b1faf99d7bada725ae2a16cff"
server_key_version: "encryptionkey@0"
I20260620 00:10:56.167711 21378 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 00:10:56.168681 21393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:56.168964 21378 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:10:56.169131 21378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "bbe84cf52d9c4f7db150d727b2b95739"
format_stamp: "Formatted at 2026-06-20 00:10:56 on dist-test-slave-grqt"
server_key: "cb19998d71b0348d8b3d6bef24281726"
server_key_iv: "effe4d7b1faf99d7bada725ae2a16cff"
server_key_version: "encryptionkey@0"
I20260620 00:10:56.169245 21378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:10:56.196081 21378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:10:56.198307 21378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:10:56.198424 21378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:10:56.202428 21378 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37247
I20260620 00:10:56.202463 21445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37247 every 8 connection(s)
I20260620 00:10:56.202838 21378 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:10:56.203760 21446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:10:56.206185 21446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbe84cf52d9c4f7db150d727b2b95739: Bootstrap starting.
I20260620 00:10:56.206735 21446 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbe84cf52d9c4f7db150d727b2b95739: Neither blocks nor log segments found. Creating new log.
I20260620 00:10:56.206974 21446 log.cc:824] T 00000000000000000000000000000000 P bbe84cf52d9c4f7db150d727b2b95739: Log is configured to *not* fsync() on all Append() calls
I20260620 00:10:56.207669 21446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbe84cf52d9c4f7db150d727b2b95739: No bootstrap required, opened a new log
I20260620 00:10:56.209247 21446 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbe84cf52d9c4f7db150d727b2b95739 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe84cf52d9c4f7db150d727b2b95739" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37247 } }
I20260620 00:10:56.209475 21446 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbe84cf52d9c4f7db150d727b2b95739 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:10:56.209506 21446 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbe84cf52d9c4f7db150d727b2b95739 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe84cf52d9c4f7db150d727b2b95739, State: Initialized, Role: FOLLOWER
I20260620 00:10:56.209625 21446 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbe84cf52d9c4f7db150d727b2b95739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe84cf52d9c4f7db150d727b2b95739" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37247 } }
I20260620 00:10:56.209678 21446 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbe84cf52d9c4f7db150d727b2b95739 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:10:56.209700 21446 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbe84cf52d9c4f7db150d727b2b95739 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:10:56.209728 21446 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbe84cf52d9c4f7db150d727b2b95739 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:10:56.210254 21446 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbe84cf52d9c4f7db150d727b2b95739 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe84cf52d9c4f7db150d727b2b95739" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37247 } }
I20260620 00:10:56.210341 21446 leader_election.cc:302] T 00000000000000000000000000000000 P bbe84cf52d9c4f7db150d727b2b95739 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbe84cf52d9c4f7db150d727b2b95739; no voters:
I20260620 00:10:56.210496 21446 leader_election.cc:288] T 00000000000000000000000000000000 P bbe84cf52d9c4f7db150d727b2b95739 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:10:56.210594 21449 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbe84cf52d9c4f7db150d727b2b95739 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:10:56.210753 21449 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbe84cf52d9c4f7db150d727b2b95739 [term 1 LEADER]: Becoming Leader. State: Replica: bbe84cf52d9c4f7db150d727b2b95739, State: Running, Role: LEADER
I20260620 00:10:56.210790 21446 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbe84cf52d9c4f7db150d727b2b95739 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:10:56.210843 21449 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbe84cf52d9c4f7db150d727b2b95739 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe84cf52d9c4f7db150d727b2b95739" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37247 } }
I20260620 00:10:56.211223 21451 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbe84cf52d9c4f7db150d727b2b95739 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbe84cf52d9c4f7db150d727b2b95739. Latest consensus state: current_term: 1 leader_uuid: "bbe84cf52d9c4f7db150d727b2b95739" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe84cf52d9c4f7db150d727b2b95739" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37247 } } }
I20260620 00:10:56.211314 21451 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbe84cf52d9c4f7db150d727b2b95739 [sys.catalog]: This master's current role is: LEADER
I20260620 00:10:56.211361 21450 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbe84cf52d9c4f7db150d727b2b95739 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbe84cf52d9c4f7db150d727b2b95739" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe84cf52d9c4f7db150d727b2b95739" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37247 } } }
I20260620 00:10:56.211431 21450 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbe84cf52d9c4f7db150d727b2b95739 [sys.catalog]: This master's current role is: LEADER
I20260620 00:10:56.211561 21456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:10:56.212113 21456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:10:56.212931 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 21378
I20260620 00:10:56.213063 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:10:56.213286 392 external_mini_cluster.cc:1428] Setting key e133b3a75b9a1ea7a11741c50e023d0c
I20260620 00:10:56.213809 21456 catalog_manager.cc:1347] Generated new cluster ID: 5c4e169c3bce4f1ca64dd656b4e1afb4
I20260620 00:10:56.213860 21456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:10:56.218266 21469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:56.213785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.57:39151 (local address 127.0.0.1:37247)
0620 00:10:56.214089 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:56.214096 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:56.214119 (+ 23us) server_negotiation.cc:407] Connection header received
0620 00:10:56.214211 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:56.214216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:56.214262 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:56.214399 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:56.214577 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:56.215477 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:56.216319 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:56.216759 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:56.216875 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:56.217642 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:56.217646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:56.217646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:10:56.217831 (+ 185us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:56.217862 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:10:56.217873 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:56.217970 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:56.218048 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 00:10:56.218077 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":78,"threads_started":1}
I20260620 00:10:56.219205 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:10:56.221823 21456 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:10:56.222314 21456 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:10:56.224146 21469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:56.220704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.57:35403 (local address 127.0.0.1:37247)
0620 00:10:56.220740 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:56.220741 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:56.220750 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:10:56.220795 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:56.220795 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:56.220805 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:56.220860 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:56.221003 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:56.221597 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:56.222783 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:56.223090 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:56.223154 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:56.223870 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:56.223871 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:56.223871 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:10:56.223875 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:56.223889 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:10:56.223893 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:56.223946 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:56.224063 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 00:10:56.224075 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 00:10:56.228215 21456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbe84cf52d9c4f7db150d727b2b95739: Generated new TSK 0
I20260620 00:10:56.228385 21456 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:10:56.233348 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 21378
W20260620 00:10:56.237227 21363 connection.cc:535] client connection to 127.0.0.1:37247 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37247): Connection reset by peer (error 104)
2026-06-20T00:10:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1339 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:10:56 dist-test-slave-grqt krb5kdc[21477](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:10:56 dist-test-slave-grqt krb5kdc[21477](info): set up 2 sockets
Jun 20 00:10:56 dist-test-slave-grqt krb5kdc[21477](info): commencing operation
krb5kdc: starting...
W20260620 00:10:57.261468 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:10:57 dist-test-slave-grqt krb5kdc[21477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:10:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:10:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 00:10:57.378448 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36531
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:10:57.462049 21490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:10:57.462286 21490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:10:57.462312 21490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:10:57.463667 21490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:10:57.463724 21490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:10:57.463739 21490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:10:57.463752 21490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:10:57.463765 21490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:10:57.465428 21490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:10:57.465719 21490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:10:57.465981 21490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:10:57.471015 21495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:57.471113 21496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:57.471180 21498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:57.471271 21490 server_base.cc:1034] running on GCE node
I20260620 00:10:57.471534 21490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:10:57.471828 21490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:10:57.472996 21490 hybrid_clock.cc:648] HybridClock initialized: now 1781914257472969 us; error 58 us; skew 500 ppm
Jun 20 00:10:57 dist-test-slave-grqt krb5kdc[21477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914257, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:10:57.475623 21490 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 00:10:57.476115 21490 webserver.cc:460] Webserver started at http://127.0.0.1:33463/ using document root <none> and password file <none>
I20260620 00:10:57.476348 21490 fs_manager.cc:359] Metadata directory not provided
I20260620 00:10:57.476414 21490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:10:57.476569 21490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:10:57.477731 21490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "43c949f511b14091a45820a5bdb4c083"
format_stamp: "Formatted at 2026-06-20 00:10:57 on dist-test-slave-grqt"
server_key: "3c1bd36089a268d432a53efcb2333264"
server_key_iv: "4041fec5863fce6ab86a0554f0dbd148"
server_key_version: "encryptionkey@0"
I20260620 00:10:57.478088 21490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "43c949f511b14091a45820a5bdb4c083"
format_stamp: "Formatted at 2026-06-20 00:10:57 on dist-test-slave-grqt"
server_key: "3c1bd36089a268d432a53efcb2333264"
server_key_iv: "4041fec5863fce6ab86a0554f0dbd148"
server_key_version: "encryptionkey@0"
I20260620 00:10:57.479418 21490 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:10:57.480281 21505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:57.480485 21490 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:10:57.480613 21490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "43c949f511b14091a45820a5bdb4c083"
format_stamp: "Formatted at 2026-06-20 00:10:57 on dist-test-slave-grqt"
server_key: "3c1bd36089a268d432a53efcb2333264"
server_key_iv: "4041fec5863fce6ab86a0554f0dbd148"
server_key_version: "encryptionkey@0"
I20260620 00:10:57.480687 21490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:10:57.484380 21490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:10:57.486845 21490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:10:57.487020 21490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:10:57.491276 21490 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36531
I20260620 00:10:57.491360 21557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36531 every 8 connection(s)
I20260620 00:10:57.491705 21490 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:10:57.492698 21558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:10:57.493141 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 21490
I20260620 00:10:57.493264 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:10:57.493515 392 external_mini_cluster.cc:1428] Setting key 1631f94aa38842fe188f14d69819184e
I20260620 00:10:57.495589 21558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43c949f511b14091a45820a5bdb4c083: Bootstrap starting.
I20260620 00:10:57.495985 21561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:57.494053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.57:54205 (local address 127.0.0.1:36531)
0620 00:10:57.494225 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:57.494233 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:57.494259 (+ 26us) server_negotiation.cc:407] Connection header received
0620 00:10:57.494682 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:57.494689 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:57.494713 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:57.494830 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:57.495528 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:57.495531 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:57.495532 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:10:57.495535 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:57.495571 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:10:57.495581 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:57.495630 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:57.495751 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 00:10:57.495799 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":82,"threads_started":1}
I20260620 00:10:57.496315 21558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43c949f511b14091a45820a5bdb4c083: Neither blocks nor log segments found. Creating new log.
I20260620 00:10:57.496636 21558 log.cc:824] T 00000000000000000000000000000000 P 43c949f511b14091a45820a5bdb4c083: Log is configured to *not* fsync() on all Append() calls
I20260620 00:10:57.497184 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:10:57.497502 21558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43c949f511b14091a45820a5bdb4c083: No bootstrap required, opened a new log
I20260620 00:10:57.499583 21561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:57.498632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.57:37983 (local address 127.0.0.1:36531)
0620 00:10:57.498669 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:57.498671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:57.498772 (+ 101us) server_negotiation.cc:407] Connection header received
0620 00:10:57.498850 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:57.498851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:57.498855 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:57.498895 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:57.499405 (+ 510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:57.499406 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:57.499406 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:10:57.499409 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:57.499420 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:10:57.499423 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:57.499462 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:57.499530 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 00:10:57.499540 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 00:10:57.499918 21558 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43c949f511b14091a45820a5bdb4c083 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c949f511b14091a45820a5bdb4c083" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36531 } }
I20260620 00:10:57.500116 21558 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43c949f511b14091a45820a5bdb4c083 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:10:57.500185 21558 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43c949f511b14091a45820a5bdb4c083 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43c949f511b14091a45820a5bdb4c083, State: Initialized, Role: FOLLOWER
I20260620 00:10:57.500312 21558 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43c949f511b14091a45820a5bdb4c083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c949f511b14091a45820a5bdb4c083" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36531 } }
I20260620 00:10:57.500429 21558 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43c949f511b14091a45820a5bdb4c083 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:10:57.500468 21558 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43c949f511b14091a45820a5bdb4c083 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:10:57.500519 21558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43c949f511b14091a45820a5bdb4c083 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:10:57.501036 21558 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43c949f511b14091a45820a5bdb4c083 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c949f511b14091a45820a5bdb4c083" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36531 } }
I20260620 00:10:57.501153 21558 leader_election.cc:302] T 00000000000000000000000000000000 P 43c949f511b14091a45820a5bdb4c083 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43c949f511b14091a45820a5bdb4c083; no voters:
I20260620 00:10:57.501338 21558 leader_election.cc:288] T 00000000000000000000000000000000 P 43c949f511b14091a45820a5bdb4c083 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:10:57.501407 21568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43c949f511b14091a45820a5bdb4c083 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:10:57.501570 21568 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43c949f511b14091a45820a5bdb4c083 [term 1 LEADER]: Becoming Leader. State: Replica: 43c949f511b14091a45820a5bdb4c083, State: Running, Role: LEADER
I20260620 00:10:57.501622 21558 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43c949f511b14091a45820a5bdb4c083 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:10:57.501653 21568 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43c949f511b14091a45820a5bdb4c083 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c949f511b14091a45820a5bdb4c083" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36531 } }
I20260620 00:10:57.502043 21570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43c949f511b14091a45820a5bdb4c083 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43c949f511b14091a45820a5bdb4c083. Latest consensus state: current_term: 1 leader_uuid: "43c949f511b14091a45820a5bdb4c083" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c949f511b14091a45820a5bdb4c083" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36531 } } }
I20260620 00:10:57.502079 21569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43c949f511b14091a45820a5bdb4c083 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43c949f511b14091a45820a5bdb4c083" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c949f511b14091a45820a5bdb4c083" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36531 } } }
I20260620 00:10:57.502149 21570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43c949f511b14091a45820a5bdb4c083 [sys.catalog]: This master's current role is: LEADER
I20260620 00:10:57.502157 21569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43c949f511b14091a45820a5bdb4c083 [sys.catalog]: This master's current role is: LEADER
I20260620 00:10:57.502548 21576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:10:57.503059 21576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:10:57.504407 21576 catalog_manager.cc:1347] Generated new cluster ID: 4034c564a50647f08ab43048d4551960
I20260620 00:10:57.504477 21576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:10:57.533216 21576 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:10:57.533447 21576 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:10:57.536468 21576 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43c949f511b14091a45820a5bdb4c083: Generated new TSK 0
I20260620 00:10:57.536671 21576 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:10:57.544261 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 21490
2026-06-20T00:10:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1310 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:10:57 dist-test-slave-grqt krb5kdc[21589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:10:57 dist-test-slave-grqt krb5kdc[21589](info): set up 2 sockets
Jun 20 00:10:57 dist-test-slave-grqt krb5kdc[21589](info): commencing operation
krb5kdc: starting...
W20260620 00:10:58.589589 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:10:58 dist-test-slave-grqt krb5kdc[21589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:10:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:10:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 00:10:58.708499 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42805
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:10:58.792698 21602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:10:58.793020 21602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:10:58.793057 21602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:10:58.794371 21602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:10:58.794420 21602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:10:58.794435 21602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:10:58.794448 21602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:10:58.794461 21602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:10:58.795959 21602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42805
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:10:58.796211 21602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:10:58.796458 21602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:10:58.801404 21607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:10:58.801460 21610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:58.801470 21602 server_base.cc:1034] running on GCE node
W20260620 00:10:58.801404 21608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:10:58.801971 21602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:10:58.802223 21602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:10:58.803360 21602 hybrid_clock.cc:648] HybridClock initialized: now 1781914258803346 us; error 28 us; skew 500 ppm
Jun 20 00:10:58 dist-test-slave-grqt krb5kdc[21589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:10:58.810766 21602 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 00:10:58.811241 21602 webserver.cc:460] Webserver started at http://127.0.0.1:37805/ using document root <none> and password file <none>
I20260620 00:10:58.811477 21602 fs_manager.cc:359] Metadata directory not provided
I20260620 00:10:58.811542 21602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:10:58.811684 21602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:10:58.812759 21602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "e29bf354e61847e282ece2736e82da6b"
format_stamp: "Formatted at 2026-06-20 00:10:58 on dist-test-slave-grqt"
server_key: "af258de817166a117439a5a45d42f3b8"
server_key_iv: "23f80b63dd95851b6772634135981447"
server_key_version: "encryptionkey@0"
I20260620 00:10:58.813110 21602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "e29bf354e61847e282ece2736e82da6b"
format_stamp: "Formatted at 2026-06-20 00:10:58 on dist-test-slave-grqt"
server_key: "af258de817166a117439a5a45d42f3b8"
server_key_iv: "23f80b63dd95851b6772634135981447"
server_key_version: "encryptionkey@0"
I20260620 00:10:58.814450 21602 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 00:10:58.815151 21617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:10:58.815315 21602 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:10:58.815405 21602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "e29bf354e61847e282ece2736e82da6b"
format_stamp: "Formatted at 2026-06-20 00:10:58 on dist-test-slave-grqt"
server_key: "af258de817166a117439a5a45d42f3b8"
server_key_iv: "23f80b63dd95851b6772634135981447"
server_key_version: "encryptionkey@0"
I20260620 00:10:58.815485 21602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:10:58.824792 21602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:10:58.826612 21602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:10:58.826768 21602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:10:58.830813 21602 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42805
I20260620 00:10:58.830844 21669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42805 every 8 connection(s)
I20260620 00:10:58.831204 21602 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:10:58.832047 21670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:10:58.833204 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 21602
I20260620 00:10:58.833340 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:10:58.833572 392 external_mini_cluster.cc:1428] Setting key 850fa7c23d3c403b5e138f8e7768d992
I20260620 00:10:58.834820 21670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e29bf354e61847e282ece2736e82da6b: Bootstrap starting.
I20260620 00:10:58.835564 21670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e29bf354e61847e282ece2736e82da6b: Neither blocks nor log segments found. Creating new log.
I20260620 00:10:58.835889 21670 log.cc:824] T 00000000000000000000000000000000 P e29bf354e61847e282ece2736e82da6b: Log is configured to *not* fsync() on all Append() calls
I20260620 00:10:58.836583 21670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e29bf354e61847e282ece2736e82da6b: No bootstrap required, opened a new log
I20260620 00:10:58.838968 21670 raft_consensus.cc:348] T 00000000000000000000000000000000 P e29bf354e61847e282ece2736e82da6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e29bf354e61847e282ece2736e82da6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42805 } }
I20260620 00:10:58.839177 21670 raft_consensus.cc:374] T 00000000000000000000000000000000 P e29bf354e61847e282ece2736e82da6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:10:58.839219 21670 raft_consensus.cc:729] T 00000000000000000000000000000000 P e29bf354e61847e282ece2736e82da6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e29bf354e61847e282ece2736e82da6b, State: Initialized, Role: FOLLOWER
I20260620 00:10:58.839357 21670 consensus_queue.cc:260] T 00000000000000000000000000000000 P e29bf354e61847e282ece2736e82da6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e29bf354e61847e282ece2736e82da6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42805 } }
I20260620 00:10:58.839445 21670 raft_consensus.cc:388] T 00000000000000000000000000000000 P e29bf354e61847e282ece2736e82da6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:10:58.839493 21670 raft_consensus.cc:482] T 00000000000000000000000000000000 P e29bf354e61847e282ece2736e82da6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:10:58.839552 21670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e29bf354e61847e282ece2736e82da6b [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:10:58.840322 21670 raft_consensus.cc:504] T 00000000000000000000000000000000 P e29bf354e61847e282ece2736e82da6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e29bf354e61847e282ece2736e82da6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42805 } }
I20260620 00:10:58.840413 21670 leader_election.cc:302] T 00000000000000000000000000000000 P e29bf354e61847e282ece2736e82da6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e29bf354e61847e282ece2736e82da6b; no voters:
I20260620 00:10:58.840607 21670 leader_election.cc:288] T 00000000000000000000000000000000 P e29bf354e61847e282ece2736e82da6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:10:58.840631 21675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e29bf354e61847e282ece2736e82da6b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:10:58.840747 21675 raft_consensus.cc:686] T 00000000000000000000000000000000 P e29bf354e61847e282ece2736e82da6b [term 1 LEADER]: Becoming Leader. State: Replica: e29bf354e61847e282ece2736e82da6b, State: Running, Role: LEADER
Jun 20 00:10:58 dist-test-slave-grqt krb5kdc[21589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 00:10:58.840876 21670 sys_catalog.cc:564] T 00000000000000000000000000000000 P e29bf354e61847e282ece2736e82da6b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:10:58.840896 21675 consensus_queue.cc:237] T 00000000000000000000000000000000 P e29bf354e61847e282ece2736e82da6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e29bf354e61847e282ece2736e82da6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42805 } }
I20260620 00:10:58.841238 21675 sys_catalog.cc:455] T 00000000000000000000000000000000 P e29bf354e61847e282ece2736e82da6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e29bf354e61847e282ece2736e82da6b. Latest consensus state: current_term: 1 leader_uuid: "e29bf354e61847e282ece2736e82da6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e29bf354e61847e282ece2736e82da6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42805 } } }
I20260620 00:10:58.841255 21676 sys_catalog.cc:455] T 00000000000000000000000000000000 P e29bf354e61847e282ece2736e82da6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e29bf354e61847e282ece2736e82da6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e29bf354e61847e282ece2736e82da6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42805 } } }
I20260620 00:10:58.841423 21676 sys_catalog.cc:458] T 00000000000000000000000000000000 P e29bf354e61847e282ece2736e82da6b [sys.catalog]: This master's current role is: LEADER
I20260620 00:10:58.841575 21675 sys_catalog.cc:458] T 00000000000000000000000000000000 P e29bf354e61847e282ece2736e82da6b [sys.catalog]: This master's current role is: LEADER
I20260620 00:10:58.842360 21682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:10:58.843132 21682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:10:58.844693 21682 catalog_manager.cc:1347] Generated new cluster ID: b2f7441b490f4f3aa43bb54faf8cc01a
I20260620 00:10:58.844743 21682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:10:58.845572 21673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:58.834131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.57:60363 (local address 127.0.0.1:42805)
0620 00:10:58.834287 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:58.834297 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:58.834327 (+ 30us) server_negotiation.cc:407] Connection header received
0620 00:10:58.834744 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:58.834750 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:58.834798 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:58.834958 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:58.835760 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:58.836843 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:58.837945 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:58.838352 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:58.838431 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:58.841606 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:58.841610 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:58.841610 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:10:58.841623 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:58.843854 (+ 2231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:58.844310 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:58.844313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:58.844313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:58.844404 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:58.844638 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:58.844639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:58.844639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:58.844943 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:58.845094 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:58.845258 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 00:10:58.845357 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
I20260620 00:10:58.846597 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:10:58.855176 21673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:10:58.848044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.57:59803 (local address 127.0.0.1:42805)
0620 00:10:58.848081 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 00:10:58.848083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:10:58.848092 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:10:58.848134 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:10:58.848135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:10:58.848144 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:10:58.848211 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:10:58.848779 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:58.849622 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:58.850595 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:10:58.850927 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:10:58.851004 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:10:58.852305 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:10:58.852306 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:10:58.852306 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:10:58.852313 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:10:58.853896 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:58.854287 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:58.854288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:58.854288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:58.854349 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:10:58.854569 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:10:58.854569 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:10:58.854570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:10:58.854701 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:10:58.854808 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 00:10:58.854912 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 00:10:58.854976 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 00:10:58.867206 21682 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:10:58.867813 21682 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:10:58.872677 21682 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e29bf354e61847e282ece2736e82da6b: Generated new TSK 0
I20260620 00:10:58.872882 21682 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:10:58.876438 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 21602
W20260620 00:10:58.880832 21587 connection.cc:535] client connection to 127.0.0.1:42805 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42805): Connection reset by peer (error 104)
2026-06-20T00:10:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1332 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:10:58 dist-test-slave-grqt krb5kdc[21701](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:10:58 dist-test-slave-grqt krb5kdc[21701](info): set up 2 sockets
Jun 20 00:10:58 dist-test-slave-grqt krb5kdc[21701](info): commencing operation
krb5kdc: starting...
W20260620 00:10:59.909922 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:10:59 dist-test-slave-grqt krb5kdc[21701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:10:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:10:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 00:11:00.028295 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33169
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:11:00.109543 21714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:11:00.109761 21714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:11:00.109786 21714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:11:00.110914 21714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:11:00.110958 21714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:11:00.110973 21714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:11:00.110985 21714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:11:00.110997 21714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:11:00.112450 21714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33169
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:11:00.112658 21714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:11:00.112836 21714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 00:11:00.117785 21714 server_base.cc:1034] running on GCE node
W20260620 00:11:00.117980 21722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:11:00.117789 21720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:11:00.117789 21719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:11:00.118317 21714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:11:00.118623 21714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:11:00.119789 21714 hybrid_clock.cc:648] HybridClock initialized: now 1781914260119772 us; error 37 us; skew 500 ppm
Jun 20 00:11:00 dist-test-slave-grqt krb5kdc[21701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914260, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:11:00.122499 21714 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 00:11:00.122991 21714 webserver.cc:460] Webserver started at http://127.0.0.1:40583/ using document root <none> and password file <none>
I20260620 00:11:00.123224 21714 fs_manager.cc:359] Metadata directory not provided
I20260620 00:11:00.123289 21714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:11:00.123432 21714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:11:00.124532 21714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "91526632d8364668bb6b6c57fdb0e593"
format_stamp: "Formatted at 2026-06-20 00:11:00 on dist-test-slave-grqt"
server_key: "53d5963fc46f2f7be68d2ac672cbe203"
server_key_iv: "5051d167628904c8ec34a09f4abc0856"
server_key_version: "encryptionkey@0"
I20260620 00:11:00.124897 21714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "91526632d8364668bb6b6c57fdb0e593"
format_stamp: "Formatted at 2026-06-20 00:11:00 on dist-test-slave-grqt"
server_key: "53d5963fc46f2f7be68d2ac672cbe203"
server_key_iv: "5051d167628904c8ec34a09f4abc0856"
server_key_version: "encryptionkey@0"
I20260620 00:11:00.126392 21714 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:11:00.127197 21729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:11:00.127415 21714 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:11:00.127534 21714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "91526632d8364668bb6b6c57fdb0e593"
format_stamp: "Formatted at 2026-06-20 00:11:00 on dist-test-slave-grqt"
server_key: "53d5963fc46f2f7be68d2ac672cbe203"
server_key_iv: "5051d167628904c8ec34a09f4abc0856"
server_key_version: "encryptionkey@0"
I20260620 00:11:00.127614 21714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:11:00.144272 21714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:11:00.146720 21714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:11:00.146903 21714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:11:00.151021 21714 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33169
I20260620 00:11:00.151087 21781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33169 every 8 connection(s)
I20260620 00:11:00.151453 21714 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:11:00.152469 21782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:11:00.153121 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 21714
I20260620 00:11:00.153237 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:11:00.153477 392 external_mini_cluster.cc:1428] Setting key 79ffbc15ee450551cca700ec58e1c829
I20260620 00:11:00.155551 21782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91526632d8364668bb6b6c57fdb0e593: Bootstrap starting.
I20260620 00:11:00.156186 21782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91526632d8364668bb6b6c57fdb0e593: Neither blocks nor log segments found. Creating new log.
I20260620 00:11:00.156523 21782 log.cc:824] T 00000000000000000000000000000000 P 91526632d8364668bb6b6c57fdb0e593: Log is configured to *not* fsync() on all Append() calls
I20260620 00:11:00.157272 21782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91526632d8364668bb6b6c57fdb0e593: No bootstrap required, opened a new log
I20260620 00:11:00.159139 21785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:00.154051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.57:33839 (local address 127.0.0.1:33169)
0620 00:11:00.154208 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:00.154218 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:00.154252 (+ 34us) server_negotiation.cc:407] Connection header received
0620 00:11:00.154679 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:00.154685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:00.154735 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:00.154870 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:00.155095 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:00.156031 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:00.157108 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:00.157597 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:00.157717 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:00.158445 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:00.158448 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:00.158448 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:11:00.158621 (+ 173us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:00.158661 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:11:00.158677 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:00.158748 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:00.158908 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 00:11:00.158943 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
I20260620 00:11:00.159654 21782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91526632d8364668bb6b6c57fdb0e593 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91526632d8364668bb6b6c57fdb0e593" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33169 } }
I20260620 00:11:00.159857 21782 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91526632d8364668bb6b6c57fdb0e593 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:11:00.159911 21782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91526632d8364668bb6b6c57fdb0e593 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91526632d8364668bb6b6c57fdb0e593, State: Initialized, Role: FOLLOWER
I20260620 00:11:00.160044 21782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91526632d8364668bb6b6c57fdb0e593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91526632d8364668bb6b6c57fdb0e593" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33169 } }
I20260620 00:11:00.160122 21782 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91526632d8364668bb6b6c57fdb0e593 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:11:00.160133 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:11:00.160161 21782 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91526632d8364668bb6b6c57fdb0e593 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:11:00.160205 21782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91526632d8364668bb6b6c57fdb0e593 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:11:00.160962 21782 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91526632d8364668bb6b6c57fdb0e593 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91526632d8364668bb6b6c57fdb0e593" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33169 } }
I20260620 00:11:00.161077 21782 leader_election.cc:302] T 00000000000000000000000000000000 P 91526632d8364668bb6b6c57fdb0e593 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91526632d8364668bb6b6c57fdb0e593; no voters:
I20260620 00:11:00.161262 21782 leader_election.cc:288] T 00000000000000000000000000000000 P 91526632d8364668bb6b6c57fdb0e593 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:11:00.161302 21789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91526632d8364668bb6b6c57fdb0e593 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:11:00.161530 21789 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91526632d8364668bb6b6c57fdb0e593 [term 1 LEADER]: Becoming Leader. State: Replica: 91526632d8364668bb6b6c57fdb0e593, State: Running, Role: LEADER
I20260620 00:11:00.161609 21782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91526632d8364668bb6b6c57fdb0e593 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:11:00.161667 21789 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91526632d8364668bb6b6c57fdb0e593 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91526632d8364668bb6b6c57fdb0e593" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33169 } }
I20260620 00:11:00.162083 21789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91526632d8364668bb6b6c57fdb0e593 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91526632d8364668bb6b6c57fdb0e593. Latest consensus state: current_term: 1 leader_uuid: "91526632d8364668bb6b6c57fdb0e593" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91526632d8364668bb6b6c57fdb0e593" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33169 } } }
I20260620 00:11:00.162213 21789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91526632d8364668bb6b6c57fdb0e593 [sys.catalog]: This master's current role is: LEADER
I20260620 00:11:00.162256 21793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91526632d8364668bb6b6c57fdb0e593 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91526632d8364668bb6b6c57fdb0e593" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91526632d8364668bb6b6c57fdb0e593" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33169 } } }
I20260620 00:11:00.162415 21793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91526632d8364668bb6b6c57fdb0e593 [sys.catalog]: This master's current role is: LEADER
I20260620 00:11:00.162497 21801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:11:00.163017 21801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:11:00.165488 21801 catalog_manager.cc:1347] Generated new cluster ID: 01e6363b7fb541faa2359ef4535c5561
I20260620 00:11:00.165567 21801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:11:00.165769 21785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:00.161905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.57:46071 (local address 127.0.0.1:33169)
0620 00:11:00.161954 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:00.161956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:00.161995 (+ 39us) server_negotiation.cc:407] Connection header received
0620 00:11:00.162098 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:00.162099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:00.162111 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:00.162184 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:00.162350 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:00.163091 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:00.164099 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:00.164407 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:00.164474 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:00.165332 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:00.165334 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:00.165334 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:11:00.165339 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:00.165354 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:11:00.165359 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:00.165432 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:00.165661 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 00:11:00.165672 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 00:11:00.180938 21801 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:11:00.181804 21801 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:11:00.190052 21801 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91526632d8364668bb6b6c57fdb0e593: Generated new TSK 0
I20260620 00:11:00.190287 21801 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:11:00.193029 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 21714
W20260620 00:11:00.198362 21699 connection.cc:535] client connection to 127.0.0.1:33169 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33169): Connection reset by peer (error 104)
2026-06-20T00:11:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1317 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:11:00 dist-test-slave-grqt krb5kdc[21813](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:11:00 dist-test-slave-grqt krb5kdc[21813](info): set up 2 sockets
Jun 20 00:11:00 dist-test-slave-grqt krb5kdc[21813](info): commencing operation
krb5kdc: starting...
W20260620 00:11:01.229588 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:11:01 dist-test-slave-grqt krb5kdc[21813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:11:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:11:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 00:11:01.352304 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45421
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:11:01.433466 21826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:11:01.433712 21826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:11:01.433748 21826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:11:01.434947 21826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:11:01.435014 21826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:11:01.435041 21826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:11:01.435072 21826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:11:01.435096 21826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:11:01.436641 21826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45421
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914001714546-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:11:01.436918 21826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:11:01.437143 21826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:11:01.442214 21834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:11:01.442184 21832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:11:01.442440 21826 server_base.cc:1034] running on GCE node
W20260620 00:11:01.442179 21831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:11:01.442721 21826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:11:01.442981 21826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:11:01.444120 21826 hybrid_clock.cc:648] HybridClock initialized: now 1781914261444106 us; error 31 us; skew 500 ppm
Jun 20 00:11:01 dist-test-slave-grqt krb5kdc[21813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:11:01.446638 21826 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 00:11:01.447220 21826 webserver.cc:460] Webserver started at http://127.0.0.1:41863/ using document root <none> and password file <none>
I20260620 00:11:01.447507 21826 fs_manager.cc:359] Metadata directory not provided
I20260620 00:11:01.447573 21826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:11:01.447692 21826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:11:01.449103 21826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "9b25ddb142754c6ab4b7163bc797e23b"
format_stamp: "Formatted at 2026-06-20 00:11:01 on dist-test-slave-grqt"
server_key: "33902269ba17f23b7d1767d4bb27262c"
server_key_iv: "593683989aa1f532a29fe94aba427929"
server_key_version: "encryptionkey@0"
I20260620 00:11:01.449496 21826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "9b25ddb142754c6ab4b7163bc797e23b"
format_stamp: "Formatted at 2026-06-20 00:11:01 on dist-test-slave-grqt"
server_key: "33902269ba17f23b7d1767d4bb27262c"
server_key_iv: "593683989aa1f532a29fe94aba427929"
server_key_version: "encryptionkey@0"
I20260620 00:11:01.451143 21826 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:11:01.452051 21841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:11:01.452257 21826 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:11:01.452375 21826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "9b25ddb142754c6ab4b7163bc797e23b"
format_stamp: "Formatted at 2026-06-20 00:11:01 on dist-test-slave-grqt"
server_key: "33902269ba17f23b7d1767d4bb27262c"
server_key_iv: "593683989aa1f532a29fe94aba427929"
server_key_version: "encryptionkey@0"
I20260620 00:11:01.452454 21826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:11:01.455858 21826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:11:01.458150 21826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:11:01.458333 21826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:11:01.462621 21826 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45421
I20260620 00:11:01.462656 21893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45421 every 8 connection(s)
I20260620 00:11:01.462987 21826 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:11:01.463977 21894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:11:01.466390 21894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b25ddb142754c6ab4b7163bc797e23b: Bootstrap starting.
I20260620 00:11:01.467037 21894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b25ddb142754c6ab4b7163bc797e23b: Neither blocks nor log segments found. Creating new log.
I20260620 00:11:01.467046 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 21826
I20260620 00:11:01.467157 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:11:01.467396 392 external_mini_cluster.cc:1428] Setting key 19ba0843903dd811573d4dfe910d0c06
I20260620 00:11:01.467386 21894 log.cc:824] T 00000000000000000000000000000000 P 9b25ddb142754c6ab4b7163bc797e23b: Log is configured to *not* fsync() on all Append() calls
I20260620 00:11:01.468170 21894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b25ddb142754c6ab4b7163bc797e23b: No bootstrap required, opened a new log
I20260620 00:11:01.469974 21898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:01.467877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.57:41731 (local address 127.0.0.1:45421)
0620 00:11:01.468094 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:01.468101 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:01.468126 (+ 25us) server_negotiation.cc:407] Connection header received
0620 00:11:01.468592 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:01.468599 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:01.468627 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:01.468743 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:01.469510 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:01.469513 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:01.469514 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:11:01.469518 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:01.469558 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:11:01.469568 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:01.469638 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:01.469713 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 00:11:01.469766 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260620 00:11:01.470991 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:11:01.470906 21894 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b25ddb142754c6ab4b7163bc797e23b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b25ddb142754c6ab4b7163bc797e23b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45421 } }
I20260620 00:11:01.471055 21894 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b25ddb142754c6ab4b7163bc797e23b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:11:01.471112 21894 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b25ddb142754c6ab4b7163bc797e23b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b25ddb142754c6ab4b7163bc797e23b, State: Initialized, Role: FOLLOWER
I20260620 00:11:01.471235 21894 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b25ddb142754c6ab4b7163bc797e23b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b25ddb142754c6ab4b7163bc797e23b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45421 } }
I20260620 00:11:01.471318 21894 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b25ddb142754c6ab4b7163bc797e23b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:11:01.471340 21894 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b25ddb142754c6ab4b7163bc797e23b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:11:01.471403 21894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b25ddb142754c6ab4b7163bc797e23b [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:11:01.471992 21894 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b25ddb142754c6ab4b7163bc797e23b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b25ddb142754c6ab4b7163bc797e23b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45421 } }
I20260620 00:11:01.472108 21894 leader_election.cc:302] T 00000000000000000000000000000000 P 9b25ddb142754c6ab4b7163bc797e23b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b25ddb142754c6ab4b7163bc797e23b; no voters:
I20260620 00:11:01.472282 21894 leader_election.cc:288] T 00000000000000000000000000000000 P 9b25ddb142754c6ab4b7163bc797e23b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:11:01.472314 21903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b25ddb142754c6ab4b7163bc797e23b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:11:01.472472 21903 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b25ddb142754c6ab4b7163bc797e23b [term 1 LEADER]: Becoming Leader. State: Replica: 9b25ddb142754c6ab4b7163bc797e23b, State: Running, Role: LEADER
I20260620 00:11:01.472550 21894 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b25ddb142754c6ab4b7163bc797e23b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:11:01.472565 21903 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b25ddb142754c6ab4b7163bc797e23b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b25ddb142754c6ab4b7163bc797e23b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45421 } }
I20260620 00:11:01.472898 21903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b25ddb142754c6ab4b7163bc797e23b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b25ddb142754c6ab4b7163bc797e23b. Latest consensus state: current_term: 1 leader_uuid: "9b25ddb142754c6ab4b7163bc797e23b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b25ddb142754c6ab4b7163bc797e23b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45421 } } }
I20260620 00:11:01.472986 21903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b25ddb142754c6ab4b7163bc797e23b [sys.catalog]: This master's current role is: LEADER
I20260620 00:11:01.472905 21906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b25ddb142754c6ab4b7163bc797e23b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b25ddb142754c6ab4b7163bc797e23b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b25ddb142754c6ab4b7163bc797e23b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45421 } } }
I20260620 00:11:01.473091 21906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b25ddb142754c6ab4b7163bc797e23b [sys.catalog]: This master's current role is: LEADER
I20260620 00:11:01.473218 21910 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:11:01.473739 21910 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:11:01.474285 21898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:01.472513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.57:52671 (local address 127.0.0.1:45421)
0620 00:11:01.472551 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:01.472553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:01.472632 (+ 79us) server_negotiation.cc:407] Connection header received
0620 00:11:01.472725 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:01.472727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:01.472760 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:01.472792 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:01.473870 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:01.473871 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:01.473872 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:11:01.473876 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:01.473889 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:11:01.473894 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:01.473937 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:01.474215 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 00:11:01.474228 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 00:11:01.475988 21910 catalog_manager.cc:1347] Generated new cluster ID: df1d6c50aafd4dbbaf191303d04c77f9
I20260620 00:11:01.476048 21910 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:11:01.496898 21910 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:11:01.497102 21910 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:11:01.504019 21910 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b25ddb142754c6ab4b7163bc797e23b: Generated new TSK 0
I20260620 00:11:01.504227 21910 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:11:01.511277 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 21826
2026-06-20T00:11:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1317 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:11:01 dist-test-slave-grqt krb5kdc[21925](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:11:01 dist-test-slave-grqt krb5kdc[21925](info): set up 2 sockets
Jun 20 00:11:01 dist-test-slave-grqt krb5kdc[21925](info): commencing operation
krb5kdc: starting...
W20260620 00:11:02.557603 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:11:02 dist-test-slave-grqt krb5kdc[21925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:11:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:11:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:11:02.676735 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32871
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:11:02.756927 21938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:11:02.757136 21938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:11:02.757158 21938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:11:02.758349 21938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:11:02.758399 21938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:11:02.758412 21938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:11:02.758425 21938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:11:02.758477 21938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:11:02.759968 21938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32871
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:11:02.760237 21938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:11:02.760473 21938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:11:02.765409 21946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:11:02.765450 21944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:11:02.765421 21943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:11:02.765578 21938 server_base.cc:1034] running on GCE node
I20260620 00:11:02.765957 21938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:11:02.766235 21938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:11:02.767377 21938 hybrid_clock.cc:648] HybridClock initialized: now 1781914262767363 us; error 29 us; skew 500 ppm
Jun 20 00:11:02 dist-test-slave-grqt krb5kdc[21925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914262, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:11:02.769912 21938 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:11:02.770315 21938 webserver.cc:460] Webserver started at http://127.0.98.62:46581/ using document root <none> and password file <none>
I20260620 00:11:02.770519 21938 fs_manager.cc:359] Metadata directory not provided
I20260620 00:11:02.770560 21938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:11:02.770648 21938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:11:02.771695 21938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "5eff56c1754042f284861d92b0cb1f44"
format_stamp: "Formatted at 2026-06-20 00:11:02 on dist-test-slave-grqt"
server_key: "0b5fc37773baeaf19e67d71c61756cb3"
server_key_iv: "5dcd35c27fab6b91776f70db25ab7c7f"
server_key_version: "encryptionkey@0"
I20260620 00:11:02.772013 21938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "5eff56c1754042f284861d92b0cb1f44"
format_stamp: "Formatted at 2026-06-20 00:11:02 on dist-test-slave-grqt"
server_key: "0b5fc37773baeaf19e67d71c61756cb3"
server_key_iv: "5dcd35c27fab6b91776f70db25ab7c7f"
server_key_version: "encryptionkey@0"
I20260620 00:11:02.773334 21938 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 00:11:02.774159 21953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:11:02.774359 21938 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:11:02.774462 21938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "5eff56c1754042f284861d92b0cb1f44"
format_stamp: "Formatted at 2026-06-20 00:11:02 on dist-test-slave-grqt"
server_key: "0b5fc37773baeaf19e67d71c61756cb3"
server_key_iv: "5dcd35c27fab6b91776f70db25ab7c7f"
server_key_version: "encryptionkey@0"
I20260620 00:11:02.774529 21938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:11:02.821630 21938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:11:02.824131 21938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:11:02.824324 21938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:11:02.828478 21938 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32871
I20260620 00:11:02.828519 22005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32871 every 8 connection(s)
I20260620 00:11:02.828950 21938 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:11:02.829954 22006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:11:02.832345 22006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eff56c1754042f284861d92b0cb1f44: Bootstrap starting.
I20260620 00:11:02.832377 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 21938
I20260620 00:11:02.832489 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:11:02.832710 392 external_mini_cluster.cc:1428] Setting key 2175e95d5990c0dbb44dfd364b5f4699
I20260620 00:11:02.833084 22006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5eff56c1754042f284861d92b0cb1f44: Neither blocks nor log segments found. Creating new log.
I20260620 00:11:02.833454 22006 log.cc:824] T 00000000000000000000000000000000 P 5eff56c1754042f284861d92b0cb1f44: Log is configured to *not* fsync() on all Append() calls
I20260620 00:11:02.834141 22006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eff56c1754042f284861d92b0cb1f44: No bootstrap required, opened a new log
I20260620 00:11:02.835925 22006 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5eff56c1754042f284861d92b0cb1f44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eff56c1754042f284861d92b0cb1f44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32871 } }
I20260620 00:11:02.836083 22006 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5eff56c1754042f284861d92b0cb1f44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:11:02.836131 22006 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5eff56c1754042f284861d92b0cb1f44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eff56c1754042f284861d92b0cb1f44, State: Initialized, Role: FOLLOWER
I20260620 00:11:02.836279 22006 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5eff56c1754042f284861d92b0cb1f44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eff56c1754042f284861d92b0cb1f44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32871 } }
I20260620 00:11:02.836357 22006 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5eff56c1754042f284861d92b0cb1f44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:11:02.836400 22006 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5eff56c1754042f284861d92b0cb1f44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:11:02.836450 22006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5eff56c1754042f284861d92b0cb1f44 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:11:02.836973 22006 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5eff56c1754042f284861d92b0cb1f44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eff56c1754042f284861d92b0cb1f44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32871 } }
I20260620 00:11:02.837126 22006 leader_election.cc:302] T 00000000000000000000000000000000 P 5eff56c1754042f284861d92b0cb1f44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5eff56c1754042f284861d92b0cb1f44; no voters:
I20260620 00:11:02.837323 22006 leader_election.cc:288] T 00000000000000000000000000000000 P 5eff56c1754042f284861d92b0cb1f44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:11:02.837348 22011 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5eff56c1754042f284861d92b0cb1f44 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:11:02.837486 22011 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5eff56c1754042f284861d92b0cb1f44 [term 1 LEADER]: Becoming Leader. State: Replica: 5eff56c1754042f284861d92b0cb1f44, State: Running, Role: LEADER
I20260620 00:11:02.837616 22006 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5eff56c1754042f284861d92b0cb1f44 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:11:02.837621 22011 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5eff56c1754042f284861d92b0cb1f44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eff56c1754042f284861d92b0cb1f44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32871 } }
I20260620 00:11:02.838013 22011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eff56c1754042f284861d92b0cb1f44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5eff56c1754042f284861d92b0cb1f44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eff56c1754042f284861d92b0cb1f44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32871 } } }
I20260620 00:11:02.838153 22011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eff56c1754042f284861d92b0cb1f44 [sys.catalog]: This master's current role is: LEADER
I20260620 00:11:02.838331 22011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eff56c1754042f284861d92b0cb1f44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5eff56c1754042f284861d92b0cb1f44. Latest consensus state: current_term: 1 leader_uuid: "5eff56c1754042f284861d92b0cb1f44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eff56c1754042f284861d92b0cb1f44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32871 } } }
I20260620 00:11:02.838449 22011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eff56c1754042f284861d92b0cb1f44 [sys.catalog]: This master's current role is: LEADER
I20260620 00:11:02.838964 22019 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:11:02.839449 22019 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 00:11:02 dist-test-slave-grqt krb5kdc[21925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:11:02.841271 22019 catalog_manager.cc:1347] Generated new cluster ID: 8d328ac927184c78a2971c6ad220453e
I20260620 00:11:02.841325 22019 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:11:02.844756 22009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:02.833231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52415 (local address 127.0.98.62:32871)
0620 00:11:02.833460 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:02.833470 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:02.833522 (+ 52us) server_negotiation.cc:407] Connection header received
0620 00:11:02.834118 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:02.834126 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:02.834195 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:02.834353 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:02.834995 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:02.835891 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:02.836861 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:02.837349 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:02.837523 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:02.840454 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:02.840458 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:02.840458 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:11:02.840469 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:02.842563 (+ 2094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:02.843055 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:02.843059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:02.843060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:02.843180 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:02.843444 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:02.843445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:02.843445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:02.843754 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:02.843919 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:02.844425 (+ 506us) server_negotiation.cc:299] Negotiation successful
0620 00:11:02.844534 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":89,"threads_started":1}
I20260620 00:11:02.845712 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:11:02.854295 22009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:02.847065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42795 (local address 127.0.98.62:32871)
0620 00:11:02.847165 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:02.847167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:02.847204 (+ 37us) server_negotiation.cc:407] Connection header received
0620 00:11:02.847263 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:02.847264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:02.847275 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:02.847339 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:02.847924 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:02.848695 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:02.849695 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:02.849995 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:02.850070 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:02.851409 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:02.851410 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:02.851410 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:11:02.851417 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:02.853007 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:02.853498 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:02.853499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:02.853499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:02.853559 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:02.853780 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:02.853781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:02.853781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:02.853913 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:02.854019 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:02.854133 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 00:11:02.854196 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 00:11:02.857594 22019 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:11:02.858137 22019 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:11:02.863018 22019 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5eff56c1754042f284861d92b0cb1f44: Generated new TSK 0
I20260620 00:11:02.863232 22019 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:11:02.866035 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 21938
W20260620 00:11:02.870599 21923 connection.cc:535] client connection to 127.0.98.62:32871 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:32871): Connection reset by peer (error 104)
2026-06-20T00:11:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1354 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:11:02 dist-test-slave-grqt krb5kdc[22037](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:11:02 dist-test-slave-grqt krb5kdc[22037](info): set up 2 sockets
Jun 20 00:11:02 dist-test-slave-grqt krb5kdc[22037](info): commencing operation
krb5kdc: starting...
W20260620 00:11:03.905560 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:11:03 dist-test-slave-grqt krb5kdc[22037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:11:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:11:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:11:04.023583 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39249
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:11:04.104379 22050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:11:04.104636 22050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:11:04.104678 22050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:11:04.105868 22050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:11:04.105935 22050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:11:04.105960 22050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:11:04.105989 22050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:11:04.106017 22050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:11:04.107482 22050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39249
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:11:04.107748 22050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:11:04.107993 22050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:11:04.112910 22055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:11:04.112919 22058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:11:04.112902 22056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:11:04.113230 22050 server_base.cc:1034] running on GCE node
I20260620 00:11:04.113520 22050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:11:04.113816 22050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:11:04.114957 22050 hybrid_clock.cc:648] HybridClock initialized: now 1781914264114939 us; error 35 us; skew 500 ppm
Jun 20 00:11:04 dist-test-slave-grqt krb5kdc[22037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914264, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:11:04.122102 22050 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:11:04.122555 22050 webserver.cc:460] Webserver started at http://127.0.98.62:39185/ using document root <none> and password file <none>
I20260620 00:11:04.122795 22050 fs_manager.cc:359] Metadata directory not provided
I20260620 00:11:04.122843 22050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:11:04.122964 22050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:11:04.124080 22050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "e782dd0e1f2a4746bf3c54ceb4050f8e"
format_stamp: "Formatted at 2026-06-20 00:11:04 on dist-test-slave-grqt"
server_key: "372b5215a4a7284e3fc3d28a9fb694eb"
server_key_iv: "6f178b67f557acf2eac5b6dff95457ad"
server_key_version: "encryptionkey@0"
I20260620 00:11:04.124436 22050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "e782dd0e1f2a4746bf3c54ceb4050f8e"
format_stamp: "Formatted at 2026-06-20 00:11:04 on dist-test-slave-grqt"
server_key: "372b5215a4a7284e3fc3d28a9fb694eb"
server_key_iv: "6f178b67f557acf2eac5b6dff95457ad"
server_key_version: "encryptionkey@0"
I20260620 00:11:04.125850 22050 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:11:04.126624 22065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:11:04.126875 22050 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:11:04.126971 22050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "e782dd0e1f2a4746bf3c54ceb4050f8e"
format_stamp: "Formatted at 2026-06-20 00:11:04 on dist-test-slave-grqt"
server_key: "372b5215a4a7284e3fc3d28a9fb694eb"
server_key_iv: "6f178b67f557acf2eac5b6dff95457ad"
server_key_version: "encryptionkey@0"
I20260620 00:11:04.127040 22050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:11:04.141433 22050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:11:04.143774 22050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:11:04.143926 22050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:11:04.147804 22050 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39249
I20260620 00:11:04.147852 22117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39249 every 8 connection(s)
I20260620 00:11:04.148217 22050 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:11:04.149058 22118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:11:04.151539 22118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e782dd0e1f2a4746bf3c54ceb4050f8e: Bootstrap starting.
I20260620 00:11:04.152107 22118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e782dd0e1f2a4746bf3c54ceb4050f8e: Neither blocks nor log segments found. Creating new log.
I20260620 00:11:04.152388 22118 log.cc:824] T 00000000000000000000000000000000 P e782dd0e1f2a4746bf3c54ceb4050f8e: Log is configured to *not* fsync() on all Append() calls
I20260620 00:11:04.153059 22118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e782dd0e1f2a4746bf3c54ceb4050f8e: No bootstrap required, opened a new log
I20260620 00:11:04.154685 22118 raft_consensus.cc:348] T 00000000000000000000000000000000 P e782dd0e1f2a4746bf3c54ceb4050f8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e782dd0e1f2a4746bf3c54ceb4050f8e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39249 } }
I20260620 00:11:04.154834 22118 raft_consensus.cc:374] T 00000000000000000000000000000000 P e782dd0e1f2a4746bf3c54ceb4050f8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:11:04.154857 22118 raft_consensus.cc:729] T 00000000000000000000000000000000 P e782dd0e1f2a4746bf3c54ceb4050f8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e782dd0e1f2a4746bf3c54ceb4050f8e, State: Initialized, Role: FOLLOWER
I20260620 00:11:04.155036 22118 consensus_queue.cc:260] T 00000000000000000000000000000000 P e782dd0e1f2a4746bf3c54ceb4050f8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e782dd0e1f2a4746bf3c54ceb4050f8e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39249 } }
I20260620 00:11:04.155115 22118 raft_consensus.cc:388] T 00000000000000000000000000000000 P e782dd0e1f2a4746bf3c54ceb4050f8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:11:04.155165 22118 raft_consensus.cc:482] T 00000000000000000000000000000000 P e782dd0e1f2a4746bf3c54ceb4050f8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:11:04.155221 22118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e782dd0e1f2a4746bf3c54ceb4050f8e [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:11:04.155715 22118 raft_consensus.cc:504] T 00000000000000000000000000000000 P e782dd0e1f2a4746bf3c54ceb4050f8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e782dd0e1f2a4746bf3c54ceb4050f8e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39249 } }
I20260620 00:11:04.155821 22118 leader_election.cc:302] T 00000000000000000000000000000000 P e782dd0e1f2a4746bf3c54ceb4050f8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e782dd0e1f2a4746bf3c54ceb4050f8e; no voters:
I20260620 00:11:04.155999 22118 leader_election.cc:288] T 00000000000000000000000000000000 P e782dd0e1f2a4746bf3c54ceb4050f8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:11:04.156041 22121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e782dd0e1f2a4746bf3c54ceb4050f8e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:11:04.156206 22121 raft_consensus.cc:686] T 00000000000000000000000000000000 P e782dd0e1f2a4746bf3c54ceb4050f8e [term 1 LEADER]: Becoming Leader. State: Replica: e782dd0e1f2a4746bf3c54ceb4050f8e, State: Running, Role: LEADER
I20260620 00:11:04.156265 22118 sys_catalog.cc:564] T 00000000000000000000000000000000 P e782dd0e1f2a4746bf3c54ceb4050f8e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:11:04.156383 22121 consensus_queue.cc:237] T 00000000000000000000000000000000 P e782dd0e1f2a4746bf3c54ceb4050f8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e782dd0e1f2a4746bf3c54ceb4050f8e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39249 } }
I20260620 00:11:04.156749 22123 sys_catalog.cc:455] T 00000000000000000000000000000000 P e782dd0e1f2a4746bf3c54ceb4050f8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e782dd0e1f2a4746bf3c54ceb4050f8e. Latest consensus state: current_term: 1 leader_uuid: "e782dd0e1f2a4746bf3c54ceb4050f8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e782dd0e1f2a4746bf3c54ceb4050f8e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39249 } } }
I20260620 00:11:04.156828 22123 sys_catalog.cc:458] T 00000000000000000000000000000000 P e782dd0e1f2a4746bf3c54ceb4050f8e [sys.catalog]: This master's current role is: LEADER
I20260620 00:11:04.156877 22122 sys_catalog.cc:455] T 00000000000000000000000000000000 P e782dd0e1f2a4746bf3c54ceb4050f8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e782dd0e1f2a4746bf3c54ceb4050f8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e782dd0e1f2a4746bf3c54ceb4050f8e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39249 } } }
I20260620 00:11:04.156937 22122 sys_catalog.cc:458] T 00000000000000000000000000000000 P e782dd0e1f2a4746bf3c54ceb4050f8e [sys.catalog]: This master's current role is: LEADER
I20260620 00:11:04.157051 22128 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:11:04.157590 22128 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:11:04.158330 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 22050
I20260620 00:11:04.158452 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:11:04.158704 392 external_mini_cluster.cc:1428] Setting key 1d01783f8e8d026415e9f8a0b59cbec1
I20260620 00:11:04.159632 22128 catalog_manager.cc:1347] Generated new cluster ID: 4134e55f2c81406480662122a8ed77a5
I20260620 00:11:04.159710 22128 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:11:04.167095 22141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:04.159223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38397 (local address 127.0.98.62:39249)
0620 00:11:04.159735 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:04.159745 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:04.159778 (+ 33us) server_negotiation.cc:407] Connection header received
0620 00:11:04.159894 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:04.159902 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:04.159959 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:04.160106 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:04.160430 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:04.161926 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:04.163002 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:04.163632 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:04.163756 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:04.165784 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:04.165788 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:04.165788 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:11:04.166140 (+ 352us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:04.166206 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:11:04.166225 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:04.166654 (+ 429us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:04.166707 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 00:11:04.166772 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":70,"threads_started":1}
I20260620 00:11:04.168699 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:11:04.176307 22141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:04.171042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39861 (local address 127.0.98.62:39249)
0620 00:11:04.171113 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:04.171115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:04.171435 (+ 320us) server_negotiation.cc:407] Connection header received
0620 00:11:04.171566 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:04.171567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:04.171600 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:04.171632 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:04.172007 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:04.173096 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:04.174726 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:04.175064 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:04.175108 (+ 44us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:04.176007 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:04.176009 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:04.176009 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:11:04.176015 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:04.176029 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:11:04.176034 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:04.176066 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:04.176200 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 00:11:04.176214 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 00:11:04.185914 22128 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:11:04.186693 22128 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:11:04.193823 22128 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e782dd0e1f2a4746bf3c54ceb4050f8e: Generated new TSK 0
I20260620 00:11:04.194039 22128 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:11:04.196738 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 22050
W20260620 00:11:04.201097 22035 connection.cc:535] client connection to 127.0.98.62:39249 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:39249): Connection reset by peer (error 104)
2026-06-20T00:11:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1330 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:11:04 dist-test-slave-grqt krb5kdc[22149](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:11:04 dist-test-slave-grqt krb5kdc[22149](info): set up 2 sockets
Jun 20 00:11:04 dist-test-slave-grqt krb5kdc[22149](info): commencing operation
krb5kdc: starting...
W20260620 00:11:05.229643 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:11:05 dist-test-slave-grqt krb5kdc[22149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:11:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:11:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:11:05.348086 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37801
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:11:05.429328 22162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:11:05.429569 22162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:11:05.429592 22162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:11:05.430735 22162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:11:05.430775 22162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:11:05.430790 22162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:11:05.430804 22162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:11:05.430816 22162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:11:05.432294 22162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37801
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:11:05.432518 22162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:11:05.432721 22162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 00:11:05.437804 22162 server_base.cc:1034] running on GCE node
W20260620 00:11:05.437846 22170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:11:05.437912 22168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:11:05.437773 22167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:11:05.438217 22162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:11:05.438514 22162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:11:05.439692 22162 hybrid_clock.cc:648] HybridClock initialized: now 1781914265439666 us; error 39 us; skew 500 ppm
Jun 20 00:11:05 dist-test-slave-grqt krb5kdc[22149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914265, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:11:05.442394 22162 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:11:05.442895 22162 webserver.cc:460] Webserver started at http://127.0.98.62:39347/ using document root <none> and password file <none>
I20260620 00:11:05.443150 22162 fs_manager.cc:359] Metadata directory not provided
I20260620 00:11:05.443218 22162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:11:05.443354 22162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:11:05.444451 22162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "b4b98500c7004891977943605b2eacf0"
format_stamp: "Formatted at 2026-06-20 00:11:05 on dist-test-slave-grqt"
server_key: "d714649dbd2b3b90829643b6b0945fe0"
server_key_iv: "d20b8b26aad6b5c1e0b3a523091ce0c4"
server_key_version: "encryptionkey@0"
I20260620 00:11:05.444825 22162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "b4b98500c7004891977943605b2eacf0"
format_stamp: "Formatted at 2026-06-20 00:11:05 on dist-test-slave-grqt"
server_key: "d714649dbd2b3b90829643b6b0945fe0"
server_key_iv: "d20b8b26aad6b5c1e0b3a523091ce0c4"
server_key_version: "encryptionkey@0"
I20260620 00:11:05.446337 22162 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:11:05.447191 22177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:11:05.447404 22162 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:11:05.447512 22162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "b4b98500c7004891977943605b2eacf0"
format_stamp: "Formatted at 2026-06-20 00:11:05 on dist-test-slave-grqt"
server_key: "d714649dbd2b3b90829643b6b0945fe0"
server_key_iv: "d20b8b26aad6b5c1e0b3a523091ce0c4"
server_key_version: "encryptionkey@0"
I20260620 00:11:05.447595 22162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:11:05.451068 22162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:11:05.453115 22162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:11:05.453297 22162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:11:05.457942 22162 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37801
I20260620 00:11:05.457974 22229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37801 every 8 connection(s)
I20260620 00:11:05.458340 22162 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:11:05.459429 22230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:11:05.462262 22230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4b98500c7004891977943605b2eacf0: Bootstrap starting.
I20260620 00:11:05.462842 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 22162
I20260620 00:11:05.462980 22230 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4b98500c7004891977943605b2eacf0: Neither blocks nor log segments found. Creating new log.
I20260620 00:11:05.462992 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:11:05.463188 392 external_mini_cluster.cc:1428] Setting key fd3e4eb7970111baa8bc699c9abe75ca
I20260620 00:11:05.463402 22230 log.cc:824] T 00000000000000000000000000000000 P b4b98500c7004891977943605b2eacf0: Log is configured to *not* fsync() on all Append() calls
I20260620 00:11:05.464473 22230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4b98500c7004891977943605b2eacf0: No bootstrap required, opened a new log
I20260620 00:11:05.467146 22230 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4b98500c7004891977943605b2eacf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4b98500c7004891977943605b2eacf0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37801 } }
I20260620 00:11:05.467326 22230 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4b98500c7004891977943605b2eacf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:11:05.467375 22230 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4b98500c7004891977943605b2eacf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4b98500c7004891977943605b2eacf0, State: Initialized, Role: FOLLOWER
I20260620 00:11:05.467521 22230 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4b98500c7004891977943605b2eacf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4b98500c7004891977943605b2eacf0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37801 } }
I20260620 00:11:05.467621 22230 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4b98500c7004891977943605b2eacf0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:11:05.467677 22230 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4b98500c7004891977943605b2eacf0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:11:05.467734 22230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4b98500c7004891977943605b2eacf0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:11:05.468616 22234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:05.463831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39219 (local address 127.0.98.62:37801)
0620 00:11:05.464520 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:05.464531 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:05.465605 (+ 1074us) server_negotiation.cc:407] Connection header received
0620 00:11:05.467140 (+ 1535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:05.467149 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:05.467179 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:05.467378 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:05.468174 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:05.468178 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:05.468178 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:11:05.468182 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:05.468230 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:11:05.468242 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:05.468316 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:05.468394 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 00:11:05.468439 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":71,"threads_started":1}
I20260620 00:11:05.468981 22230 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4b98500c7004891977943605b2eacf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4b98500c7004891977943605b2eacf0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37801 } }
I20260620 00:11:05.469120 22230 leader_election.cc:302] T 00000000000000000000000000000000 P b4b98500c7004891977943605b2eacf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4b98500c7004891977943605b2eacf0; no voters:
I20260620 00:11:05.469337 22230 leader_election.cc:288] T 00000000000000000000000000000000 P b4b98500c7004891977943605b2eacf0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:11:05.469404 22235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4b98500c7004891977943605b2eacf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:11:05.469552 22235 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4b98500c7004891977943605b2eacf0 [term 1 LEADER]: Becoming Leader. State: Replica: b4b98500c7004891977943605b2eacf0, State: Running, Role: LEADER
I20260620 00:11:05.469630 22235 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4b98500c7004891977943605b2eacf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4b98500c7004891977943605b2eacf0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37801 } }
I20260620 00:11:05.469693 22230 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4b98500c7004891977943605b2eacf0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:11:05.469832 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:11:05.469944 22235 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4b98500c7004891977943605b2eacf0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4b98500c7004891977943605b2eacf0. Latest consensus state: current_term: 1 leader_uuid: "b4b98500c7004891977943605b2eacf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4b98500c7004891977943605b2eacf0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37801 } } }
I20260620 00:11:05.470012 22235 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4b98500c7004891977943605b2eacf0 [sys.catalog]: This master's current role is: LEADER
I20260620 00:11:05.470904 22239 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:11:05.471467 22239 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:11:05.471463 22236 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4b98500c7004891977943605b2eacf0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4b98500c7004891977943605b2eacf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4b98500c7004891977943605b2eacf0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37801 } } }
I20260620 00:11:05.472487 22236 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4b98500c7004891977943605b2eacf0 [sys.catalog]: This master's current role is: LEADER
I20260620 00:11:05.473204 22239 catalog_manager.cc:1347] Generated new cluster ID: 051e53ab07ac4d6eafd5af937854386e
I20260620 00:11:05.473260 22239 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:11:05.473347 22234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:05.471606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40729 (local address 127.0.98.62:37801)
0620 00:11:05.472119 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:05.472121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:05.472128 (+ 7us) server_negotiation.cc:407] Connection header received
0620 00:11:05.472143 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:05.472144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:05.472147 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:05.472214 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:05.473074 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:05.473076 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:05.473076 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:11:05.473080 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:05.473094 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:11:05.473098 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:05.473143 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:05.473232 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 00:11:05.473248 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497}
I20260620 00:11:05.489713 22239 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:11:05.489916 22239 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:11:05.503609 22239 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4b98500c7004891977943605b2eacf0: Generated new TSK 0
I20260620 00:11:05.503835 22239 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:11:05.507664 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 22162
2026-06-20T00:11:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1310 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:11:05 dist-test-slave-grqt krb5kdc[22261](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:11:05 dist-test-slave-grqt krb5kdc[22261](info): set up 2 sockets
Jun 20 00:11:05 dist-test-slave-grqt krb5kdc[22261](info): commencing operation
krb5kdc: starting...
W20260620 00:11:06.549554 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:11:06 dist-test-slave-grqt krb5kdc[22261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:11:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:11:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:11:06.667922 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40151
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:11:06.748729 22274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:11:06.748952 22274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:11:06.748975 22274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:11:06.750113 22274 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:11:06.750149 22274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:11:06.750164 22274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:11:06.750176 22274 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:11:06.750188 22274 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:11:06.751574 22274 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:11:06.751768 22274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:11:06.751940 22274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 00:11:06.756764 22274 server_base.cc:1034] running on GCE node
W20260620 00:11:06.756721 22280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:11:06.756731 22279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:11:06.756696 22282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:11:06.757126 22274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:11:06.757421 22274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:11:06.758589 22274 hybrid_clock.cc:648] HybridClock initialized: now 1781914266758574 us; error 37 us; skew 500 ppm
Jun 20 00:11:06 dist-test-slave-grqt krb5kdc[22261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:11:06.761175 22274 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:11:06.761682 22274 webserver.cc:460] Webserver started at http://127.0.98.62:38457/ using document root <none> and password file <none>
I20260620 00:11:06.761945 22274 fs_manager.cc:359] Metadata directory not provided
I20260620 00:11:06.761991 22274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:11:06.762125 22274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:11:06.763231 22274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "8e114aa5cafe41e181b0ff53b3190089"
format_stamp: "Formatted at 2026-06-20 00:11:06 on dist-test-slave-grqt"
server_key: "5ad2e53871767e108cf3f763ab1ffed2"
server_key_iv: "9beeb622390f3ce9876e79dd596e7302"
server_key_version: "encryptionkey@0"
I20260620 00:11:06.763597 22274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "8e114aa5cafe41e181b0ff53b3190089"
format_stamp: "Formatted at 2026-06-20 00:11:06 on dist-test-slave-grqt"
server_key: "5ad2e53871767e108cf3f763ab1ffed2"
server_key_iv: "9beeb622390f3ce9876e79dd596e7302"
server_key_version: "encryptionkey@0"
I20260620 00:11:06.765055 22274 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 00:11:06.765982 22289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:11:06.766228 22274 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:11:06.766353 22274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "8e114aa5cafe41e181b0ff53b3190089"
format_stamp: "Formatted at 2026-06-20 00:11:06 on dist-test-slave-grqt"
server_key: "5ad2e53871767e108cf3f763ab1ffed2"
server_key_iv: "9beeb622390f3ce9876e79dd596e7302"
server_key_version: "encryptionkey@0"
I20260620 00:11:06.766444 22274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:11:06.781018 22274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:11:06.783447 22274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:11:06.783638 22274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:11:06.787887 22274 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40151
I20260620 00:11:06.787925 22341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40151 every 8 connection(s)
I20260620 00:11:06.788301 22274 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:11:06.789301 22342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:11:06.791712 22342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e114aa5cafe41e181b0ff53b3190089: Bootstrap starting.
I20260620 00:11:06.792250 22342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e114aa5cafe41e181b0ff53b3190089: Neither blocks nor log segments found. Creating new log.
I20260620 00:11:06.792506 22342 log.cc:824] T 00000000000000000000000000000000 P 8e114aa5cafe41e181b0ff53b3190089: Log is configured to *not* fsync() on all Append() calls
I20260620 00:11:06.792873 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 22274
I20260620 00:11:06.793005 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:11:06.793217 392 external_mini_cluster.cc:1428] Setting key 70f8cf125b5c543aa6d9dd498135d4f8
I20260620 00:11:06.793344 22342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e114aa5cafe41e181b0ff53b3190089: No bootstrap required, opened a new log
I20260620 00:11:06.796087 22342 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e114aa5cafe41e181b0ff53b3190089 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e114aa5cafe41e181b0ff53b3190089" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40151 } }
I20260620 00:11:06.796241 22342 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e114aa5cafe41e181b0ff53b3190089 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:11:06.796264 22342 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e114aa5cafe41e181b0ff53b3190089 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e114aa5cafe41e181b0ff53b3190089, State: Initialized, Role: FOLLOWER
I20260620 00:11:06.796418 22342 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e114aa5cafe41e181b0ff53b3190089 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e114aa5cafe41e181b0ff53b3190089" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40151 } }
I20260620 00:11:06.796494 22342 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e114aa5cafe41e181b0ff53b3190089 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:11:06.796515 22342 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e114aa5cafe41e181b0ff53b3190089 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:11:06.796582 22342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e114aa5cafe41e181b0ff53b3190089 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:11:06.797221 22342 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e114aa5cafe41e181b0ff53b3190089 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e114aa5cafe41e181b0ff53b3190089" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40151 } }
I20260620 00:11:06.797323 22342 leader_election.cc:302] T 00000000000000000000000000000000 P 8e114aa5cafe41e181b0ff53b3190089 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e114aa5cafe41e181b0ff53b3190089; no voters:
I20260620 00:11:06.797559 22342 leader_election.cc:288] T 00000000000000000000000000000000 P 8e114aa5cafe41e181b0ff53b3190089 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:11:06.797590 22347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e114aa5cafe41e181b0ff53b3190089 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:11:06.797708 22347 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e114aa5cafe41e181b0ff53b3190089 [term 1 LEADER]: Becoming Leader. State: Replica: 8e114aa5cafe41e181b0ff53b3190089, State: Running, Role: LEADER
I20260620 00:11:06.797799 22347 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e114aa5cafe41e181b0ff53b3190089 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e114aa5cafe41e181b0ff53b3190089" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40151 } }
I20260620 00:11:06.797861 22342 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e114aa5cafe41e181b0ff53b3190089 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:11:06.798123 22348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e114aa5cafe41e181b0ff53b3190089 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e114aa5cafe41e181b0ff53b3190089" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e114aa5cafe41e181b0ff53b3190089" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40151 } } }
I20260620 00:11:06.798231 22348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e114aa5cafe41e181b0ff53b3190089 [sys.catalog]: This master's current role is: LEADER
I20260620 00:11:06.798413 22349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e114aa5cafe41e181b0ff53b3190089 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e114aa5cafe41e181b0ff53b3190089. Latest consensus state: current_term: 1 leader_uuid: "8e114aa5cafe41e181b0ff53b3190089" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e114aa5cafe41e181b0ff53b3190089" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40151 } } }
I20260620 00:11:06.798470 22349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e114aa5cafe41e181b0ff53b3190089 [sys.catalog]: This master's current role is: LEADER
I20260620 00:11:06.798569 22352 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:11:06.799114 22352 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 00:11:06 dist-test-slave-grqt krb5kdc[22261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:11:06.802166 22352 catalog_manager.cc:1347] Generated new cluster ID: 7a79e511c7cf4dd38ac8504b0ec21ee6
I20260620 00:11:06.802227 22352 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:11:06.806440 22346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:06.793777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59357 (local address 127.0.98.62:40151)
0620 00:11:06.793969 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:06.793978 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:06.794010 (+ 32us) server_negotiation.cc:407] Connection header received
0620 00:11:06.794914 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:06.794923 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:06.794979 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:06.795108 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:06.796046 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:06.797514 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:06.798801 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:06.799500 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:06.799626 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:06.802641 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:06.802644 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:06.802645 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:11:06.802657 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:06.804715 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:06.805160 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:06.805163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:06.805163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:06.805265 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:06.805522 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:06.805523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:06.805523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:06.805819 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:06.805978 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:06.806093 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 00:11:06.806228 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":100,"threads_started":1}
I20260620 00:11:06.807473 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:11:06.807964 22352 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:11:06.808435 22352 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:11:06.815884 22352 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e114aa5cafe41e181b0ff53b3190089: Generated new TSK 0
I20260620 00:11:06.816084 22352 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:11:06.816423 22346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:06.808980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56017 (local address 127.0.98.62:40151)
0620 00:11:06.809021 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:06.809023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:06.809148 (+ 125us) server_negotiation.cc:407] Connection header received
0620 00:11:06.809216 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:06.809217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:06.809228 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:06.809278 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:06.809939 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:06.810667 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:06.811746 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:06.812057 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:06.812115 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:06.813561 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:06.813563 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:06.813563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:11:06.813570 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:06.815128 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:06.815519 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:06.815520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:06.815521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:06.815587 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:06.815781 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:06.815782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:06.815782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:06.815897 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:06.816000 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:06.816235 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 00:11:06.816323 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 00:11:06.818431 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 22274
2026-06-20T00:11:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1310 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:11:06 dist-test-slave-grqt krb5kdc[22373](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:11:06 dist-test-slave-grqt krb5kdc[22373](info): set up 2 sockets
Jun 20 00:11:06 dist-test-slave-grqt krb5kdc[22373](info): commencing operation
krb5kdc: starting...
W20260620 00:11:07.893532 392 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:11:07 dist-test-slave-grqt krb5kdc[22373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:11:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:11:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:11:08.012771 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35881
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35881
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:11:08.093962 22386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:11:08.094202 22386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:11:08.094240 22386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:11:08.095391 22386 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:11:08.095460 22386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:11:08.095485 22386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:11:08.095515 22386 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:11:08.095544 22386 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:11:08.097014 22386 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:11:08.097306 22386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:11:08.097571 22386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:11:08.102360 22394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:11:08.102361 22392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:11:08.102366 22391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:11:08.102730 22386 server_base.cc:1034] running on GCE node
I20260620 00:11:08.102939 22386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:11:08.103184 22386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:11:08.104321 22386 hybrid_clock.cc:648] HybridClock initialized: now 1781914268104314 us; error 36 us; skew 500 ppm
Jun 20 00:11:08 dist-test-slave-grqt krb5kdc[22373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:11:08.106818 22386 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:11:08.107319 22386 webserver.cc:460] Webserver started at http://127.0.98.62:37353/ using document root <none> and password file <none>
I20260620 00:11:08.107553 22386 fs_manager.cc:359] Metadata directory not provided
I20260620 00:11:08.107617 22386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:11:08.107769 22386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:11:08.108822 22386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "bc9dfe8cd91e4453996b4a7dbecf0e5e"
format_stamp: "Formatted at 2026-06-20 00:11:08 on dist-test-slave-grqt"
server_key: "6687dc29ef6085133c57d2a563985089"
server_key_iv: "dd7e8a4e89ce2b05ae3febb728656fa0"
server_key_version: "encryptionkey@0"
I20260620 00:11:08.109189 22386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "bc9dfe8cd91e4453996b4a7dbecf0e5e"
format_stamp: "Formatted at 2026-06-20 00:11:08 on dist-test-slave-grqt"
server_key: "6687dc29ef6085133c57d2a563985089"
server_key_iv: "dd7e8a4e89ce2b05ae3febb728656fa0"
server_key_version: "encryptionkey@0"
I20260620 00:11:08.110569 22386 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:11:08.111363 22401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:11:08.111557 22386 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:11:08.111656 22386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "bc9dfe8cd91e4453996b4a7dbecf0e5e"
format_stamp: "Formatted at 2026-06-20 00:11:08 on dist-test-slave-grqt"
server_key: "6687dc29ef6085133c57d2a563985089"
server_key_iv: "dd7e8a4e89ce2b05ae3febb728656fa0"
server_key_version: "encryptionkey@0"
I20260620 00:11:08.111739 22386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:11:08.126986 22386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:11:08.128851 22386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:11:08.129016 22386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:11:08.132808 22386 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35881
I20260620 00:11:08.132859 22453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35881 every 8 connection(s)
I20260620 00:11:08.133214 22386 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:11:08.134068 22454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:11:08.136377 22454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc9dfe8cd91e4453996b4a7dbecf0e5e: Bootstrap starting.
I20260620 00:11:08.136984 22454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc9dfe8cd91e4453996b4a7dbecf0e5e: Neither blocks nor log segments found. Creating new log.
I20260620 00:11:08.137252 22454 log.cc:824] T 00000000000000000000000000000000 P bc9dfe8cd91e4453996b4a7dbecf0e5e: Log is configured to *not* fsync() on all Append() calls
I20260620 00:11:08.137737 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 22386
I20260620 00:11:08.137866 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:11:08.138042 392 external_mini_cluster.cc:1428] Setting key 4cadf603c54aaf39167df88f49b27aa3
I20260620 00:11:08.138065 22454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc9dfe8cd91e4453996b4a7dbecf0e5e: No bootstrap required, opened a new log
I20260620 00:11:08.139914 22454 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc9dfe8cd91e4453996b4a7dbecf0e5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc9dfe8cd91e4453996b4a7dbecf0e5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35881 } }
I20260620 00:11:08.140081 22454 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc9dfe8cd91e4453996b4a7dbecf0e5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:11:08.140147 22454 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc9dfe8cd91e4453996b4a7dbecf0e5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc9dfe8cd91e4453996b4a7dbecf0e5e, State: Initialized, Role: FOLLOWER
I20260620 00:11:08.140321 22454 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc9dfe8cd91e4453996b4a7dbecf0e5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc9dfe8cd91e4453996b4a7dbecf0e5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35881 } }
I20260620 00:11:08.140412 22454 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc9dfe8cd91e4453996b4a7dbecf0e5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:11:08.140455 22454 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc9dfe8cd91e4453996b4a7dbecf0e5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:11:08.140503 22454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc9dfe8cd91e4453996b4a7dbecf0e5e [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:11:08.141027 22454 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc9dfe8cd91e4453996b4a7dbecf0e5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc9dfe8cd91e4453996b4a7dbecf0e5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35881 } }
I20260620 00:11:08.141136 22454 leader_election.cc:302] T 00000000000000000000000000000000 P bc9dfe8cd91e4453996b4a7dbecf0e5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc9dfe8cd91e4453996b4a7dbecf0e5e; no voters:
I20260620 00:11:08.141340 22454 leader_election.cc:288] T 00000000000000000000000000000000 P bc9dfe8cd91e4453996b4a7dbecf0e5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:11:08.141386 22459 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc9dfe8cd91e4453996b4a7dbecf0e5e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:11:08.141499 22459 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc9dfe8cd91e4453996b4a7dbecf0e5e [term 1 LEADER]: Becoming Leader. State: Replica: bc9dfe8cd91e4453996b4a7dbecf0e5e, State: Running, Role: LEADER
I20260620 00:11:08.141590 22459 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc9dfe8cd91e4453996b4a7dbecf0e5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc9dfe8cd91e4453996b4a7dbecf0e5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35881 } }
I20260620 00:11:08.141631 22454 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc9dfe8cd91e4453996b4a7dbecf0e5e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:11:08.141953 22459 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc9dfe8cd91e4453996b4a7dbecf0e5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc9dfe8cd91e4453996b4a7dbecf0e5e. Latest consensus state: current_term: 1 leader_uuid: "bc9dfe8cd91e4453996b4a7dbecf0e5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc9dfe8cd91e4453996b4a7dbecf0e5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35881 } } }
I20260620 00:11:08.141964 22460 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc9dfe8cd91e4453996b4a7dbecf0e5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc9dfe8cd91e4453996b4a7dbecf0e5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc9dfe8cd91e4453996b4a7dbecf0e5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35881 } } }
I20260620 00:11:08.142032 22459 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc9dfe8cd91e4453996b4a7dbecf0e5e [sys.catalog]: This master's current role is: LEADER
I20260620 00:11:08.142086 22460 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc9dfe8cd91e4453996b4a7dbecf0e5e [sys.catalog]: This master's current role is: LEADER
I20260620 00:11:08.142515 22465 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:11:08.143069 22465 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:11:08.144520 22458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:08.138542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56213 (local address 127.0.98.62:35881)
0620 00:11:08.138732 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:08.138742 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:08.138773 (+ 31us) server_negotiation.cc:407] Connection header received
0620 00:11:08.139283 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:08.139289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:08.139337 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:08.139480 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:08.139703 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:08.140997 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:08.142142 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:08.142782 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:08.142865 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:08.143798 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:08.143802 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:08.143802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:11:08.143973 (+ 171us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:08.144001 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:11:08.144011 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:08.144041 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:08.144240 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 00:11:08.144310 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":75,"threads_started":1}
I20260620 00:11:08.145483 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:11:08.145862 22465 catalog_manager.cc:1347] Generated new cluster ID: 7b704fc47a724acab5fba94c10d6e822
I20260620 00:11:08.145920 22465 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:11:08.152208 22458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:08.146888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36383 (local address 127.0.98.62:35881)
0620 00:11:08.146922 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:08.146924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:08.149555 (+ 2631us) server_negotiation.cc:407] Connection header received
0620 00:11:08.149660 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:08.149661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:08.149672 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:08.149729 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:08.149863 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:08.150391 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:08.151051 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:08.151260 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:08.151338 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:08.151956 (+ 618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:08.151958 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:08.151958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:11:08.151962 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:08.151975 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:11:08.151979 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:08.152032 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:08.152094 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 00:11:08.152114 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 00:11:08.157254 22465 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:11:08.157958 22465 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:11:08.161827 22465 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc9dfe8cd91e4453996b4a7dbecf0e5e: Generated new TSK 0
I20260620 00:11:08.162009 22465 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:11:08.165449 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 22386
2026-06-20T00:11:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1351 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:11:08 dist-test-slave-grqt krb5kdc[22485](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:11:08 dist-test-slave-grqt krb5kdc[22485](info): set up 2 sockets
Jun 20 00:11:08 dist-test-slave-grqt krb5kdc[22485](info): commencing operation
krb5kdc: starting...
W20260620 00:11:09.213603 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:11:09 dist-test-slave-grqt krb5kdc[22485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:11:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:11:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:11:09.334431 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39253
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:11:09.415336 22498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:11:09.415551 22498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:11:09.415575 22498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:11:09.416788 22498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:11:09.416836 22498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:11:09.416851 22498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:11:09.416863 22498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:11:09.416877 22498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:11:09.418390 22498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:11:09.418658 22498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:11:09.418890 22498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:11:09.423695 22506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:11:09.423683 22503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:11:09.423784 22504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:11:09.424098 22498 server_base.cc:1034] running on GCE node
I20260620 00:11:09.424367 22498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:11:09.424661 22498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:11:09.425854 22498 hybrid_clock.cc:648] HybridClock initialized: now 1781914269425836 us; error 35 us; skew 500 ppm
Jun 20 00:11:09 dist-test-slave-grqt krb5kdc[22485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:11:09.434458 22498 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:11:09.435127 22498 webserver.cc:460] Webserver started at http://127.0.98.62:45301/ using document root <none> and password file <none>
I20260620 00:11:09.435324 22498 fs_manager.cc:359] Metadata directory not provided
I20260620 00:11:09.435367 22498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:11:09.435461 22498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:11:09.436507 22498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "5f19f60617f4454dbef6bb197818ca58"
format_stamp: "Formatted at 2026-06-20 00:11:09 on dist-test-slave-grqt"
server_key: "8e4354f2696fcb524788d5d6d9944b0f"
server_key_iv: "10c94af2eb4907c9372fc8a319a77d13"
server_key_version: "encryptionkey@0"
I20260620 00:11:09.436826 22498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "5f19f60617f4454dbef6bb197818ca58"
format_stamp: "Formatted at 2026-06-20 00:11:09 on dist-test-slave-grqt"
server_key: "8e4354f2696fcb524788d5d6d9944b0f"
server_key_iv: "10c94af2eb4907c9372fc8a319a77d13"
server_key_version: "encryptionkey@0"
I20260620 00:11:09.438230 22498 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:11:09.439003 22513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:11:09.439206 22498 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:11:09.439299 22498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "5f19f60617f4454dbef6bb197818ca58"
format_stamp: "Formatted at 2026-06-20 00:11:09 on dist-test-slave-grqt"
server_key: "8e4354f2696fcb524788d5d6d9944b0f"
server_key_iv: "10c94af2eb4907c9372fc8a319a77d13"
server_key_version: "encryptionkey@0"
I20260620 00:11:09.439357 22498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:11:09.442698 22498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:11:09.444587 22498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:11:09.444689 22498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:11:09.448347 22498 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39253
I20260620 00:11:09.448405 22565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39253 every 8 connection(s)
I20260620 00:11:09.448736 22498 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:11:09.449146 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 22498
I20260620 00:11:09.449294 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:11:09.449579 392 external_mini_cluster.cc:1428] Setting key a4697ed84345e1786da2fffcf3be6125
I20260620 00:11:09.449978 22566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:11:09.451933 22569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:09.450097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60575 (local address 127.0.98.62:39253)
0620 00:11:09.450278 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:09.450285 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:09.450310 (+ 25us) server_negotiation.cc:407] Connection header received
0620 00:11:09.450717 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:09.450723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:09.450746 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:09.450861 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:09.451485 (+ 624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:09.451488 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:09.451489 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:11:09.451492 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:09.451530 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:11:09.451540 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:09.451593 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:09.451682 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 00:11:09.451716 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":95,"threads_started":1}
I20260620 00:11:09.453104 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:11:09.453505 22566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f19f60617f4454dbef6bb197818ca58: Bootstrap starting.
I20260620 00:11:09.454253 22566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f19f60617f4454dbef6bb197818ca58: Neither blocks nor log segments found. Creating new log.
I20260620 00:11:09.454617 22566 log.cc:824] T 00000000000000000000000000000000 P 5f19f60617f4454dbef6bb197818ca58: Log is configured to *not* fsync() on all Append() calls
I20260620 00:11:09.455432 22566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f19f60617f4454dbef6bb197818ca58: No bootstrap required, opened a new log
I20260620 00:11:09.455742 22569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:09.454676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45439 (local address 127.0.98.62:39253)
0620 00:11:09.454747 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:09.454748 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:09.454858 (+ 110us) server_negotiation.cc:407] Connection header received
0620 00:11:09.454923 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:09.454924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:09.454929 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:09.454995 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:09.455505 (+ 510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:09.455506 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:09.455506 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:11:09.455510 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:09.455525 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:11:09.455529 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:09.455578 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:09.455642 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 00:11:09.455652 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 00:11:09.457340 22566 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f19f60617f4454dbef6bb197818ca58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f19f60617f4454dbef6bb197818ca58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39253 } }
I20260620 00:11:09.457547 22566 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f19f60617f4454dbef6bb197818ca58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:11:09.457576 22566 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f19f60617f4454dbef6bb197818ca58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f19f60617f4454dbef6bb197818ca58, State: Initialized, Role: FOLLOWER
I20260620 00:11:09.457661 22566 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f19f60617f4454dbef6bb197818ca58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f19f60617f4454dbef6bb197818ca58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39253 } }
I20260620 00:11:09.457727 22566 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f19f60617f4454dbef6bb197818ca58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:11:09.457749 22566 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f19f60617f4454dbef6bb197818ca58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:11:09.457777 22566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f19f60617f4454dbef6bb197818ca58 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:11:09.458258 22566 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f19f60617f4454dbef6bb197818ca58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f19f60617f4454dbef6bb197818ca58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39253 } }
I20260620 00:11:09.458340 22566 leader_election.cc:302] T 00000000000000000000000000000000 P 5f19f60617f4454dbef6bb197818ca58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f19f60617f4454dbef6bb197818ca58; no voters:
I20260620 00:11:09.458549 22566 leader_election.cc:288] T 00000000000000000000000000000000 P 5f19f60617f4454dbef6bb197818ca58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:11:09.458593 22576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f19f60617f4454dbef6bb197818ca58 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:11:09.458750 22576 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f19f60617f4454dbef6bb197818ca58 [term 1 LEADER]: Becoming Leader. State: Replica: 5f19f60617f4454dbef6bb197818ca58, State: Running, Role: LEADER
I20260620 00:11:09.458799 22566 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f19f60617f4454dbef6bb197818ca58 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:11:09.458915 22576 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f19f60617f4454dbef6bb197818ca58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f19f60617f4454dbef6bb197818ca58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39253 } }
I20260620 00:11:09.459297 22578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f19f60617f4454dbef6bb197818ca58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f19f60617f4454dbef6bb197818ca58. Latest consensus state: current_term: 1 leader_uuid: "5f19f60617f4454dbef6bb197818ca58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f19f60617f4454dbef6bb197818ca58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39253 } } }
I20260620 00:11:09.459424 22578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f19f60617f4454dbef6bb197818ca58 [sys.catalog]: This master's current role is: LEADER
I20260620 00:11:09.459293 22577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f19f60617f4454dbef6bb197818ca58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f19f60617f4454dbef6bb197818ca58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f19f60617f4454dbef6bb197818ca58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39253 } } }
I20260620 00:11:09.459587 22577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f19f60617f4454dbef6bb197818ca58 [sys.catalog]: This master's current role is: LEADER
I20260620 00:11:09.459923 22585 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:11:09.460330 22585 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:11:09.461876 22585 catalog_manager.cc:1347] Generated new cluster ID: 838d39f79e6a465fbcb8d29ced1e938b
I20260620 00:11:09.462002 22585 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:11:09.481554 22585 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:11:09.481750 22585 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:11:09.484707 22585 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f19f60617f4454dbef6bb197818ca58: Generated new TSK 0
I20260620 00:11:09.484956 22585 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:11:09.491180 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 22498
2026-06-20T00:11:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1321 ms)
[----------] 18 tests from AuthTokenIssuingTest (23834 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:11:09 dist-test-slave-grqt krb5kdc[22597](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:11:09 dist-test-slave-grqt krb5kdc[22597](info): set up 2 sockets
Jun 20 00:11:09 dist-test-slave-grqt krb5kdc[22597](info): commencing operation
krb5kdc: starting...
W20260620 00:11:10.521673 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:11:10 dist-test-slave-grqt krb5kdc[22597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:11:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:11:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:11:10.649158 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38007
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38007,127.0.98.61:46017,127.0.98.60:37035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:11:10.731436 22610 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 00:11:10.731694 22610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:11:10.731734 22610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:11:10.731774 22610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:11:10.732952 22610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:11:10.733021 22610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:11:10.733052 22610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:11:10.733079 22610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:11:10.733103 22610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:11:10.734668 22610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38007,127.0.98.61:46017,127.0.98.60:37035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:11:10.734942 22610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:11:10.735169 22610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:11:10.739900 22615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:11:10.739923 22616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:11:10.739923 22618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:11:10.740594 22610 server_base.cc:1034] running on GCE node
I20260620 00:11:10.740851 22610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:11:10.741140 22610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:11:10.742316 22610 hybrid_clock.cc:648] HybridClock initialized: now 1781914270742304 us; error 44 us; skew 500 ppm
Jun 20 00:11:10 dist-test-slave-grqt krb5kdc[22597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:11:10.744849 22610 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:11:10.745316 22610 webserver.cc:460] Webserver started at http://127.0.98.62:46255/ using document root <none> and password file <none>
I20260620 00:11:10.745568 22610 fs_manager.cc:359] Metadata directory not provided
I20260620 00:11:10.745646 22610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:11:10.745797 22610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:11:10.746870 22610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "3d0e99784b0544c19502ed9b532443e9"
format_stamp: "Formatted at 2026-06-20 00:11:10 on dist-test-slave-grqt"
server_key: "e215171abe839107cc3a9ef0be9500c6"
server_key_iv: "93b439ece774598b3f13c90318150731"
server_key_version: "encryptionkey@0"
I20260620 00:11:10.747244 22610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "3d0e99784b0544c19502ed9b532443e9"
format_stamp: "Formatted at 2026-06-20 00:11:10 on dist-test-slave-grqt"
server_key: "e215171abe839107cc3a9ef0be9500c6"
server_key_iv: "93b439ece774598b3f13c90318150731"
server_key_version: "encryptionkey@0"
I20260620 00:11:10.748566 22610 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:11:10.749338 22625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:11:10.749548 22610 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:11:10.749655 22610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "3d0e99784b0544c19502ed9b532443e9"
format_stamp: "Formatted at 2026-06-20 00:11:10 on dist-test-slave-grqt"
server_key: "e215171abe839107cc3a9ef0be9500c6"
server_key_iv: "93b439ece774598b3f13c90318150731"
server_key_version: "encryptionkey@0"
I20260620 00:11:10.749752 22610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:11:10.764644 22610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:11:10.766781 22610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:11:10.766955 22610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:11:10.771008 22610 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38007
I20260620 00:11:10.771102 22677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38007 every 8 connection(s)
I20260620 00:11:10.771461 22610 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:11:10.772517 22678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:11:10.774124 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 22610
I20260620 00:11:10.774236 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:11:10.774431 392 external_mini_cluster.cc:1428] Setting key c83f3d3094a9bb2de610b4da94bf2aec
I20260620 00:11:10.774439 22678 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38007 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 00:11:10 dist-test-slave-grqt krb5kdc[22597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260620 00:11:10.785991 22681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:10.775572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40064 (local address 127.0.98.62:38007)
0620 00:11:10.775727 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:10.775732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:10.775771 (+ 39us) server_negotiation.cc:407] Connection header received
0620 00:11:10.776194 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:10.776196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:10.776222 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:10.776319 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:10.776968 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:10.777928 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:10.779373 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:10.779749 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:10.779849 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:10.782524 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:10.782531 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:10.782531 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:11:10.782539 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:10.784262 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:10.784692 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:10.784695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:10.784696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:10.784783 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:10.785009 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:10.785010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:10.785010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:10.785254 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:10.785350 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:10.785554 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 00:11:10.785732 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":120,"thread_start_us":67,"threads_started":1}
I20260620 00:11:10.785989 22680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:10.775297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38007 (local address 127.0.0.1:40064)
0620 00:11:10.775479 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:10.775525 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 00:11:10.776027 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:11:10.776324 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:11:10.776327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:11:10.776620 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:11:10.776777 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:11:10.776781 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:11:10.778043 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:11:10.778045 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:11:10.779267 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:11:10.779271 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:11:10.779914 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:11:10.779916 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:11:10.780011 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:10.780753 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:11:10.780764 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:11:10.782413 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:11:10.784387 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:11:10.784394 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:11:10.784396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:11:10.784596 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:11:10.784851 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:11:10.784852 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:11:10.784853 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:11:10.784943 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:11:10.785326 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:11:10.785331 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:11:10.785423 (+ 92us) client_negotiation.cc:769] Sending connection context
0620 00:11:10.785523 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 00:11:10.785770 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":108,"thread_start_us":101,"threads_started":1}
I20260620 00:11:10.787590 22678 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46017 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:11:10.787992 22680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:10.787883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46017 (local address 127.0.0.1:41018)
0620 00:11:10.787930 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:10.787972 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 00:11:10.788133 22627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46017: connect: Connection refused (error 111)
I20260620 00:11:10.788553 22680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:10.788462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46017 (local address 127.0.0.1:41032)
0620 00:11:10.788506 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:10.788529 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 00:11:10.788652 22678 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46017: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46017: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 00:11:10.801319 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46017
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38007,127.0.98.61:46017,127.0.98.60:37035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:11:10.837909 22678 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46017 } attempt: 1
I20260620 00:11:10.838438 22680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:10.838296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46017 (local address 127.0.0.1:41048)
0620 00:11:10.838362 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:10.838406 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 00:11:10.838789 22680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:10.838688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46017 (local address 127.0.0.1:41056)
0620 00:11:10.838738 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:10.838770 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 00:11:10.838871 22678 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46017: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46017: connect: Connection refused (error 111)
W20260620 00:11:10.883370 22685 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 00:11:10.883653 22685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:11:10.883702 22685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:11:10.883745 22685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:11:10.884946 22685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:11:10.885026 22685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:11:10.885052 22685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:11:10.885084 22685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:11:10.885113 22685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:11:10.886797 22685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38007,127.0.98.61:46017,127.0.98.60:37035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:11:10.887105 22685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:11:10.887367 22685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:11:10.892650 22690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:11:10.892652 22693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:11:10.892872 22685 server_base.cc:1034] running on GCE node
W20260620 00:11:10.892652 22691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:11:10.893178 22685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:11:10.893581 22685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:11:10.894760 22685 hybrid_clock.cc:648] HybridClock initialized: now 1781914270894748 us; error 39 us; skew 500 ppm
Jun 20 00:11:10 dist-test-slave-grqt krb5kdc[22597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:11:10.897568 22685 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 00:11:10.898097 22685 webserver.cc:460] Webserver started at http://127.0.98.61:42735/ using document root <none> and password file <none>
I20260620 00:11:10.898367 22685 fs_manager.cc:359] Metadata directory not provided
I20260620 00:11:10.898440 22685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:11:10.898597 22685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:11:10.899709 22685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-1/data/instance:
uuid: "81ce1a59c0ab4663b2fce47847e4c429"
format_stamp: "Formatted at 2026-06-20 00:11:10 on dist-test-slave-grqt"
server_key: "6962cd335384c79a6ba0ab457dfce8f2"
server_key_iv: "552f4387aba716d9ca48647038c2810b"
server_key_version: "encryptionkey@0"
I20260620 00:11:10.900089 22685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-1/wal/instance:
uuid: "81ce1a59c0ab4663b2fce47847e4c429"
format_stamp: "Formatted at 2026-06-20 00:11:10 on dist-test-slave-grqt"
server_key: "6962cd335384c79a6ba0ab457dfce8f2"
server_key_iv: "552f4387aba716d9ca48647038c2810b"
server_key_version: "encryptionkey@0"
I20260620 00:11:10.901618 22685 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260620 00:11:10.902464 22700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:11:10.902678 22685 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:11:10.902793 22685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-1/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-1/wal
uuid: "81ce1a59c0ab4663b2fce47847e4c429"
format_stamp: "Formatted at 2026-06-20 00:11:10 on dist-test-slave-grqt"
server_key: "6962cd335384c79a6ba0ab457dfce8f2"
server_key_iv: "552f4387aba716d9ca48647038c2810b"
server_key_version: "encryptionkey@0"
I20260620 00:11:10.902877 22685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:11:10.907020 22678 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46017 } attempt: 2
I20260620 00:11:10.907452 22680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:10.907328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46017 (local address 127.0.0.1:41064)
0620 00:11:10.907387 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:10.907425 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 00:11:10.907913 22680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:10.907789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46017 (local address 127.0.0.1:41070)
0620 00:11:10.907855 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:10.907889 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 00:11:10.908032 22678 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46017: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46017: connect: Connection refused (error 111)
I20260620 00:11:10.913192 22685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:11:10.915581 22685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:11:10.915793 22685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:11:10.920442 22685 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46017
I20260620 00:11:10.920482 22752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46017 every 8 connection(s)
I20260620 00:11:10.921070 22685 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-1/data/info.pb
I20260620 00:11:10.922154 22753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:11:10.923748 22753 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38007 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:11:10.926126 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 22685
I20260620 00:11:10.926262 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-1/wal/instance
I20260620 00:11:10.926440 392 external_mini_cluster.cc:1428] Setting key 4348e71979aeedb0418a816f57d6c2d8
Jun 20 00:11:10 dist-test-slave-grqt krb5kdc[22597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:11:10.933562 22681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:10.924489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40074 (local address 127.0.98.62:38007)
0620 00:11:10.924549 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:10.924551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:10.924774 (+ 223us) server_negotiation.cc:407] Connection header received
0620 00:11:10.925297 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:10.925298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:10.925314 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:10.925392 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:10.926044 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:10.926677 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:10.927663 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:10.927886 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:10.927974 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:10.930533 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:10.930534 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:10.930535 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:11:10.930543 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:10.932090 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:10.932535 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:10.932537 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:10.932537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:10.932598 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:10.932863 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:10.932864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:10.932865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:10.932985 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:10.933063 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:10.933359 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 00:11:10.933466 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260620 00:11:10.933780 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:10.924462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38007 (local address 127.0.0.1:40074)
0620 00:11:10.924649 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:10.924691 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 00:11:10.925173 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:11:10.925465 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:11:10.925472 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:11:10.925734 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:11:10.925927 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:11:10.925930 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:11:10.926779 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:11:10.926780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:11:10.927570 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:11:10.927572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:11:10.927999 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:11:10.928000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:11:10.928132 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:10.928743 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:11:10.928754 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:11:10.930430 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:11:10.932206 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:11:10.932214 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:11:10.932216 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:11:10.932449 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:11:10.932666 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:11:10.932667 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:11:10.932668 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:11:10.932793 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:11:10.933057 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:11:10.933065 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:11:10.933216 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 00:11:10.933328 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 00:11:10.933521 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":86,"threads_started":1}
I20260620 00:11:10.934365 22753 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46017 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 20 00:11:10 dist-test-slave-grqt krb5kdc[22597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 00:11:10.942559 22757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:10.934795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41086 (local address 127.0.98.61:46017)
0620 00:11:10.934973 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:10.934977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:10.934999 (+ 22us) server_negotiation.cc:407] Connection header received
0620 00:11:10.935017 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:10.935018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:10.935029 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:10.935112 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:10.935386 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:10.936088 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:10.937038 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:10.937342 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:10.937459 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:10.939428 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:10.939436 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:10.939437 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:11:10.939444 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:10.941198 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:10.941765 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:10.941768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:10.941768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:10.941823 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:10.942039 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:10.942040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:10.942040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:10.942173 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:10.942237 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:10.942408 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 00:11:10.942481 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":154,"thread_start_us":102,"threads_started":1}
I20260620 00:11:10.942693 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:10.934713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46017 (local address 127.0.0.1:41086)
0620 00:11:10.934753 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:10.934765 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:11:10.934822 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:11:10.935113 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:11:10.935113 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:11:10.935263 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:11:10.935308 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:11:10.935310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:11:10.936172 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:11:10.936173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:11:10.936942 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:11:10.936945 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:11:10.937471 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:11:10.937472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:11:10.937538 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:10.938015 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:11:10.938021 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:11:10.939346 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:11:10.941296 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:11:10.941298 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:11:10.941298 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:11:10.941666 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:11:10.941894 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:11:10.941895 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:11:10.941895 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:11:10.941971 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:11:10.942244 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:11:10.942246 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:11:10.942303 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 00:11:10.942387 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 00:11:10.942561 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20,"mutex_wait_us":65}
I20260620 00:11:10.943353 22753 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37035 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:11:10.943732 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:10.943594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37035 (local address 127.0.0.1:41430)
0620 00:11:10.943646 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:10.943696 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 00:11:10.943887 22701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37035: connect: Connection refused (error 111)
I20260620 00:11:10.944370 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:10.944248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37035 (local address 127.0.0.1:41446)
0620 00:11:10.944307 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:10.944339 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 00:11:10.944490 22753 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37035: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37035: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 00:11:10.952937 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37035
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38007,127.0.98.61:46017,127.0.98.60:37035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:11:10.993734 22753 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37035 } attempt: 1
I20260620 00:11:10.994279 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:10.994124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37035 (local address 127.0.0.1:41458)
0620 00:11:10.994197 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:10.994244 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 00:11:10.994668 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:10.994565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37035 (local address 127.0.0.1:41472)
0620 00:11:10.994614 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:10.994643 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 00:11:10.994799 22753 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37035: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37035: connect: Connection refused (error 111)
I20260620 00:11:10.999187 22678 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46017 } attempt: 3
Jun 20 00:11:10 dist-test-slave-grqt krb5kdc[22597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 00:11:11.006528 22680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:10.999526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46017 (local address 127.0.0.1:41098)
0620 00:11:10.999587 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:10.999597 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:11:10.999704 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:11:10.999884 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:11:10.999885 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:11:11.000047 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:11:11.000116 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:11:11.000117 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:11:11.000792 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:11:11.000794 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:11:11.001311 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:11:11.001313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:11:11.001771 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:11:11.001773 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:11:11.001858 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:11.002332 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:11:11.002337 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:11:11.003743 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:11:11.005418 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:11:11.005419 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:11:11.005420 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:11:11.005616 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:11:11.005836 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:11:11.005836 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:11:11.005837 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:11:11.005906 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:11:11.006184 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:11:11.006185 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:11:11.006235 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 00:11:11.006322 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 00:11:11.006426 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 00:11:11.006528 22757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:10.999583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41098 (local address 127.0.98.61:46017)
0620 00:11:10.999646 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:10.999648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:10.999687 (+ 39us) server_negotiation.cc:407] Connection header received
0620 00:11:10.999784 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:10.999785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:10.999797 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:10.999877 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:11.000211 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:11.000699 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:11.001449 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:11.001683 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:11.001771 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:11.003834 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:11.003835 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:11.003835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:11:11.003844 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:11.005282 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:11.005702 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:11.005703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:11.005703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:11.005765 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:11.005979 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:11.005980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:11.005980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:11.006102 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:11.006195 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:11.006345 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 00:11:11.006426 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 00:11:11.007179 22678 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37035 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:11:11.007637 22680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:11.007495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37035 (local address 127.0.0.1:41476)
0620 00:11:11.007573 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:11.007610 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 00:11:11.008172 22680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:11.008068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37035 (local address 127.0.0.1:41482)
0620 00:11:11.008115 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:11.008144 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 00:11:11.008276 22678 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37035: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37035: connect: Connection refused (error 111)
W20260620 00:11:11.037034 22760 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 00:11:11.037256 22760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:11:11.037274 22760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:11:11.037295 22760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:11:11.038514 22760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:11:11.038558 22760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:11:11.038573 22760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:11:11.038610 22760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:11:11.038637 22760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:11:11.039419 22678 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37035 } attempt: 1
I20260620 00:11:11.039861 22680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:11.039714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37035 (local address 127.0.0.1:41486)
0620 00:11:11.039783 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:11.039830 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 00:11:11.040257 22680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:11.040156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37035 (local address 127.0.0.1:41494)
0620 00:11:11.040211 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:11.040237 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 00:11:11.040364 22678 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37035: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37035: connect: Connection refused (error 111)
I20260620 00:11:11.040537 22760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38007,127.0.98.61:46017,127.0.98.60:37035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:11:11.040804 22760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:11:11.041049 22760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:11:11.046211 22768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:11:11.046182 22766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:11:11.046324 22760 server_base.cc:1034] running on GCE node
W20260620 00:11:11.046229 22765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:11:11.046622 22760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:11:11.046921 22760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:11:11.048101 22760 hybrid_clock.cc:648] HybridClock initialized: now 1781914271048072 us; error 44 us; skew 500 ppm
Jun 20 00:11:11 dist-test-slave-grqt krb5kdc[22597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:11:11.050737 22760 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 00:11:11.051191 22760 webserver.cc:460] Webserver started at http://127.0.98.60:42485/ using document root <none> and password file <none>
I20260620 00:11:11.051430 22760 fs_manager.cc:359] Metadata directory not provided
I20260620 00:11:11.051476 22760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:11:11.051651 22760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:11:11.052699 22760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-2/data/instance:
uuid: "41a40a25b08f4c129ed0d73f40fc97de"
format_stamp: "Formatted at 2026-06-20 00:11:11 on dist-test-slave-grqt"
server_key: "0fd305499c4304d6ea1ff3dcaca69dfd"
server_key_iv: "de11f7c76dc4bfa9534a545622ff94ae"
server_key_version: "encryptionkey@0"
I20260620 00:11:11.053061 22760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-2/wal/instance:
uuid: "41a40a25b08f4c129ed0d73f40fc97de"
format_stamp: "Formatted at 2026-06-20 00:11:11 on dist-test-slave-grqt"
server_key: "0fd305499c4304d6ea1ff3dcaca69dfd"
server_key_iv: "de11f7c76dc4bfa9534a545622ff94ae"
server_key_version: "encryptionkey@0"
I20260620 00:11:11.054484 22760 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:11:11.055238 22775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:11:11.055426 22760 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:11:11.055537 22760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-2/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-2/wal
uuid: "41a40a25b08f4c129ed0d73f40fc97de"
format_stamp: "Formatted at 2026-06-20 00:11:11 on dist-test-slave-grqt"
server_key: "0fd305499c4304d6ea1ff3dcaca69dfd"
server_key_iv: "de11f7c76dc4bfa9534a545622ff94ae"
server_key_version: "encryptionkey@0"
I20260620 00:11:11.055619 22760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:11:11.062953 22753 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37035 } attempt: 2
I20260620 00:11:11.063417 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:11.063291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37035 (local address 127.0.0.1:41506)
0620 00:11:11.063347 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:11.063387 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 00:11:11.063768 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:11.063689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37035 (local address 127.0.0.1:41508)
0620 00:11:11.063730 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:11.063753 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 00:11:11.063881 22753 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37035: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37035: connect: Connection refused (error 111)
I20260620 00:11:11.086961 22760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:11:11.089179 22760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:11:11.089346 22760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:11:11.093343 22760 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37035
I20260620 00:11:11.093398 22827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37035 every 8 connection(s)
I20260620 00:11:11.093808 22760 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-2/data/info.pb
I20260620 00:11:11.094879 22828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:11:11.096530 22828 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38007 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:11:11.098112 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 22760
I20260620 00:11:11.098224 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914001714546-392-0/minicluster-data/master-2/wal/instance
I20260620 00:11:11.098446 392 external_mini_cluster.cc:1428] Setting key 25f92f63b6692efcc035d9f6868cb7d7
Jun 20 00:11:11 dist-test-slave-grqt krb5kdc[22597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 00:11:11 dist-test-slave-grqt krb5kdc[22597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 00:11:11.107527 22830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:11.097305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38007 (local address 127.0.0.1:40088)
0620 00:11:11.097533 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:11.097570 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 00:11:11.098010 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:11:11.098280 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:11:11.098310 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:11:11.098564 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:11:11.098897 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:11:11.098903 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:11:11.099723 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:11:11.099724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:11:11.100882 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:11:11.100884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:11:11.101503 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:11:11.101504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:11:11.101620 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:11.102253 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:11:11.102262 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:11:11.103920 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:11:11.105843 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:11:11.105852 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:11:11.105854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:11:11.106168 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:11:11.106410 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:11:11.106410 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:11:11.106411 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:11:11.106519 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:11:11.106789 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:11:11.106796 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:11:11.106946 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 00:11:11.107057 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 00:11:11.107241 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":150,"threads_started":1}
I20260620 00:11:11.107606 22681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:11.097346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40088 (local address 127.0.98.62:38007)
0620 00:11:11.097429 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:11.097446 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:11.097672 (+ 226us) server_negotiation.cc:407] Connection header received
0620 00:11:11.098138 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:11.098140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:11.098152 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:11.098204 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:11.099018 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:11.099635 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:11.100986 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:11.101295 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:11.101422 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:11.104015 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:11.104016 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:11.104017 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:11:11.104023 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:11.105753 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:11.106256 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:11.106257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:11.106258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:11.106321 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:11.106587 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:11.106588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:11.106588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:11.106719 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:11.106798 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:11.107423 (+ 625us) server_negotiation.cc:299] Negotiation successful
0620 00:11:11.107500 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 00:11:11.108206 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:11.098841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41516 (local address 127.0.98.60:37035)
0620 00:11:11.099025 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:11.099029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:11.099125 (+ 96us) server_negotiation.cc:407] Connection header received
0620 00:11:11.099147 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:11.099149 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:11.099177 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:11.099258 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:11.099868 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:11.100985 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:11.101919 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:11.102339 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:11.102399 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:11.104808 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:11.104816 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:11.104816 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:11:11.104824 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:11.106608 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:11.107062 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:11.107066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:11.107067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:11.107127 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:11.107341 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:11.107342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:11.107343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:11.107498 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:11.107931 (+ 433us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:11.107998 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 00:11:11.108097 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":120,"threads_started":1}
I20260620 00:11:11.108322 22828 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46017 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 00:11:11 dist-test-slave-grqt krb5kdc[22597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 00:11:11 dist-test-slave-grqt krb5kdc[22597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:11:11.115516 22678 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37035 } attempt: 2
I20260620 00:11:11.116739 22830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:11.108600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46017 (local address 127.0.0.1:41110)
0620 00:11:11.108740 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:11.108751 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:11:11.108797 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:11:11.108924 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:11:11.108925 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:11:11.109036 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:11:11.109086 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:11:11.109087 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:11:11.109976 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:11:11.109977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:11:11.110700 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:11:11.110702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:11:11.111208 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:11:11.111209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:11:11.111262 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:11.111752 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:11:11.111757 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:11:11.112944 (+ 1187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:11:11.115503 (+ 2559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:11:11.115504 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:11:11.115505 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:11:11.115746 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:11:11.116005 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:11:11.116006 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:11:11.116007 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:11:11.116070 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:11:11.116377 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:11:11.116378 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:11:11.116426 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 00:11:11.116532 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 00:11:11.116631 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260620 00:11:11.116770 22757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:11.108685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41110 (local address 127.0.98.61:46017)
0620 00:11:11.108743 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:11.108745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:11.108784 (+ 39us) server_negotiation.cc:407] Connection header received
0620 00:11:11.108853 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:11.108854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:11.108864 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:11.108915 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:11.109137 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:11.109903 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:11.110793 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:11.111124 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:11.111202 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:11.113036 (+ 1834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:11.113038 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:11.113038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:11:11.113046 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:11.115354 (+ 2308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:11.115866 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:11.115867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:11.115867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:11.115927 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:11.116127 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:11.116128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:11.116128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:11.116289 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:11.116350 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:11.116589 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 00:11:11.116670 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 00:11:11.117185 22681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:11.109040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40104 (local address 127.0.98.62:38007)
0620 00:11:11.109074 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:11.109076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:11.109082 (+ 6us) server_negotiation.cc:407] Connection header received
0620 00:11:11.109100 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:11.109101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:11.109108 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:11.109154 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:11.109790 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:11.110448 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:11.111391 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:11.111630 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:11.111751 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:11.114076 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:11.114077 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:11.114078 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:11:11.114084 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:11.115743 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:11.116201 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:11.116202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:11.116203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:11.116269 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:11.116500 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:11.116501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:11.116502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:11.116635 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:11.116808 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:11.117028 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 00:11:11.117102 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 00:11:11.117764 22828 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37035 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 00:11:11 dist-test-slave-grqt krb5kdc[22597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 00:11:11 dist-test-slave-grqt krb5kdc[22597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 00:11:11.123291 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:11.115873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41526 (local address 127.0.98.60:37035)
0620 00:11:11.115933 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:11.115935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:11.115942 (+ 7us) server_negotiation.cc:407] Connection header received
0620 00:11:11.115960 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:11.115961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:11.115971 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:11.116044 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:11.116335 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:11.117034 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:11.118012 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:11.118338 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:11.118432 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:11.120469 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:11.120470 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:11.120470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:11:11.120477 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:11.122138 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:11.122503 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:11.122504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:11.122505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:11.122557 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:11.122761 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:11.122762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:11.122762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:11.122873 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:11.122947 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:11.123111 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 00:11:11.123204 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 00:11:11.123291 22680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:11.115804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37035 (local address 127.0.0.1:41526)
0620 00:11:11.115843 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:11.115854 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:11:11.115886 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:11:11.116048 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:11:11.116049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:11:11.116201 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:11:11.116250 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:11:11.116252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:11:11.117120 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:11:11.117122 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:11:11.117930 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:11:11.117933 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:11:11.118436 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:11:11.118437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:11:11.118506 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:11.118934 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:11:11.118939 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:11:11.120381 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:11:11.122242 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:11:11.122242 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:11:11.122243 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:11:11.122417 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:11:11.122631 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:11:11.122632 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:11:11.122632 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:11:11.122690 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:11:11.122939 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:11:11.122940 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:11:11.122984 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 00:11:11.123084 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 00:11:11.123193 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
Jun 20 00:11:11 dist-test-slave-grqt krb5kdc[22597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 00:11:11.125638 22678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d0e99784b0544c19502ed9b532443e9: Bootstrap starting.
I20260620 00:11:11.126021 22830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:11.118022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37035 (local address 127.0.0.1:41542)
0620 00:11:11.118198 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:11.118208 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:11:11.118247 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:11:11.118372 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:11:11.118373 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:11:11.118529 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:11:11.118562 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:11:11.118564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:11:11.119289 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:11:11.119290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:11:11.120015 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:11:11.120017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:11:11.120501 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:11:11.120502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:11:11.120547 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:11.121149 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:11:11.121154 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:11:11.122454 (+ 1300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:11:11.124819 (+ 2365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:11:11.124820 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:11:11.124820 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:11:11.125028 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:11:11.125249 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:11:11.125249 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:11:11.125250 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:11:11.125314 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:11:11.125666 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:11:11.125667 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:11:11.125715 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 00:11:11.125801 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 00:11:11.125904 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20260620 00:11:11.126200 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:11.118070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41542 (local address 127.0.98.60:37035)
0620 00:11:11.118258 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:11.118260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:11.118268 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:11:11.118293 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:11.118294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:11.118305 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:11.118372 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:11.118639 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:11.119220 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:11.120106 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:11.120403 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:11.120461 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:11.122549 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:11.122550 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:11.122550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:11:11.122556 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:11.124716 (+ 2160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:11.125119 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:11.125119 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:11.125120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:11.125173 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:11.125405 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:11.125405 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:11.125406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:11.125574 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:11.125665 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:11.125841 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 00:11:11.125955 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":165,"thread_start_us":102,"threads_started":1}
I20260620 00:11:11.126448 22678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d0e99784b0544c19502ed9b532443e9: Neither blocks nor log segments found. Creating new log.
I20260620 00:11:11.126766 22678 log.cc:824] T 00000000000000000000000000000000 P 3d0e99784b0544c19502ed9b532443e9: Log is configured to *not* fsync() on all Append() calls
I20260620 00:11:11.127625 22757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:11.118588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41124 (local address 127.0.98.61:46017)
0620 00:11:11.118625 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:11.118627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:11.118635 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:11:11.119010 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:11.119011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:11.119020 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:11.119075 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:11.119648 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:11.120115 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:11.120898 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:11.121154 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:11.121219 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:11.124503 (+ 3284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:11.124504 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:11.124504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:11:11.124511 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:11.126255 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:11.126630 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:11.126632 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:11.126632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:11.126699 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:11.126911 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:11.126912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:11.126912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:11.127079 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:11.127251 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:11.127437 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 00:11:11.127515 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 00:11:11.127694 22828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41a40a25b08f4c129ed0d73f40fc97de: Bootstrap starting.
I20260620 00:11:11.127910 22678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d0e99784b0544c19502ed9b532443e9: No bootstrap required, opened a new log
I20260620 00:11:11.128367 22828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41a40a25b08f4c129ed0d73f40fc97de: Neither blocks nor log segments found. Creating new log.
I20260620 00:11:11.128641 22828 log.cc:824] T 00000000000000000000000000000000 P 41a40a25b08f4c129ed0d73f40fc97de: Log is configured to *not* fsync() on all Append() calls
I20260620 00:11:11.128818 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:11:11.129659 22828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41a40a25b08f4c129ed0d73f40fc97de: No bootstrap required, opened a new log
I20260620 00:11:11.129875 22652 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3d0e99784b0544c19502ed9b532443e9"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40104
I20260620 00:11:11.130399 22678 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d0e99784b0544c19502ed9b532443e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0e99784b0544c19502ed9b532443e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38007 } } peers { permanent_uuid: "81ce1a59c0ab4663b2fce47847e4c429" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46017 } } peers { permanent_uuid: "41a40a25b08f4c129ed0d73f40fc97de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37035 } }
I20260620 00:11:11.130535 22678 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d0e99784b0544c19502ed9b532443e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0e99784b0544c19502ed9b532443e9, State: Initialized, Role: FOLLOWER
I20260620 00:11:11.130669 22678 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d0e99784b0544c19502ed9b532443e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0e99784b0544c19502ed9b532443e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38007 } } peers { permanent_uuid: "81ce1a59c0ab4663b2fce47847e4c429" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46017 } } peers { permanent_uuid: "41a40a25b08f4c129ed0d73f40fc97de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37035 } }
I20260620 00:11:11.130949 22836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d0e99784b0544c19502ed9b532443e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0e99784b0544c19502ed9b532443e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38007 } } peers { permanent_uuid: "81ce1a59c0ab4663b2fce47847e4c429" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46017 } } peers { permanent_uuid: "41a40a25b08f4c129ed0d73f40fc97de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37035 } } }
I20260620 00:11:11.131017 22678 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d0e99784b0544c19502ed9b532443e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:11:11.131026 22836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d0e99784b0544c19502ed9b532443e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:11:11.131067 22652 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d0e99784b0544c19502ed9b532443e9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 00:11:11.131116 22652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d0e99784b0544c19502ed9b532443e9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:11:11.131870 22828 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41a40a25b08f4c129ed0d73f40fc97de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0e99784b0544c19502ed9b532443e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38007 } } peers { permanent_uuid: "81ce1a59c0ab4663b2fce47847e4c429" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46017 } } peers { permanent_uuid: "41a40a25b08f4c129ed0d73f40fc97de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37035 } }
I20260620 00:11:11.131986 22652 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d0e99784b0544c19502ed9b532443e9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0e99784b0544c19502ed9b532443e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38007 } } peers { permanent_uuid: "81ce1a59c0ab4663b2fce47847e4c429" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46017 } } peers { permanent_uuid: "41a40a25b08f4c129ed0d73f40fc97de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37035 } }
I20260620 00:11:11.132030 22828 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41a40a25b08f4c129ed0d73f40fc97de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41a40a25b08f4c129ed0d73f40fc97de, State: Initialized, Role: FOLLOWER
I20260620 00:11:11.132117 22828 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41a40a25b08f4c129ed0d73f40fc97de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0e99784b0544c19502ed9b532443e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38007 } } peers { permanent_uuid: "81ce1a59c0ab4663b2fce47847e4c429" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46017 } } peers { permanent_uuid: "41a40a25b08f4c129ed0d73f40fc97de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37035 } }
I20260620 00:11:11.132217 22652 leader_election.cc:288] T 00000000000000000000000000000000 P 3d0e99784b0544c19502ed9b532443e9 [CANDIDATE]: Term 1 election: Requested vote from peers 81ce1a59c0ab4663b2fce47847e4c429 (127.0.98.61:46017), 41a40a25b08f4c129ed0d73f40fc97de (127.0.98.60:37035)
W20260620 00:11:11.133127 22847 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3d0e99784b0544c19502ed9b532443e9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:11:11.133196 22847 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:11:11.133229 22802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3d0e99784b0544c19502ed9b532443e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "41a40a25b08f4c129ed0d73f40fc97de"
I20260620 00:11:11.133338 22802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41a40a25b08f4c129ed0d73f40fc97de [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:11:11.134301 22802 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 41a40a25b08f4c129ed0d73f40fc97de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d0e99784b0544c19502ed9b532443e9 in term 1.
I20260620 00:11:11.133160 22828 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41a40a25b08f4c129ed0d73f40fc97de [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:11:11.134544 22626 leader_election.cc:302] T 00000000000000000000000000000000 P 3d0e99784b0544c19502ed9b532443e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d0e99784b0544c19502ed9b532443e9, 41a40a25b08f4c129ed0d73f40fc97de; no voters:
I20260620 00:11:11.134693 22836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d0e99784b0544c19502ed9b532443e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:11:11.134828 22836 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d0e99784b0544c19502ed9b532443e9 [term 1 LEADER]: Becoming Leader. State: Replica: 3d0e99784b0544c19502ed9b532443e9, State: Running, Role: LEADER
I20260620 00:11:11.134940 22836 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d0e99784b0544c19502ed9b532443e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0e99784b0544c19502ed9b532443e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38007 } } peers { permanent_uuid: "81ce1a59c0ab4663b2fce47847e4c429" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46017 } } peers { permanent_uuid: "41a40a25b08f4c129ed0d73f40fc97de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37035 } }
I20260620 00:11:11.135444 22836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d0e99784b0544c19502ed9b532443e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d0e99784b0544c19502ed9b532443e9. Latest consensus state: current_term: 1 leader_uuid: "3d0e99784b0544c19502ed9b532443e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0e99784b0544c19502ed9b532443e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38007 } } peers { permanent_uuid: "81ce1a59c0ab4663b2fce47847e4c429" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46017 } } peers { permanent_uuid: "41a40a25b08f4c129ed0d73f40fc97de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37035 } } }
I20260620 00:11:11.135556 22836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d0e99784b0544c19502ed9b532443e9 [sys.catalog]: This master's current role is: LEADER
I20260620 00:11:11.135808 22866 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:11:11.136020 22866 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:11:11.134217 22610 master_runner.cc:186] Error getting master registration for 127.0.98.61:46017: OK, instance_id { permanent_uuid: "81ce1a59c0ab4663b2fce47847e4c429" instance_seqno: 1781914270918814 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 00:11:11.136519 22610 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 00:11:11.136874 22850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41a40a25b08f4c129ed0d73f40fc97de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0e99784b0544c19502ed9b532443e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38007 } } peers { permanent_uuid: "81ce1a59c0ab4663b2fce47847e4c429" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46017 } } peers { permanent_uuid: "41a40a25b08f4c129ed0d73f40fc97de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37035 } } }
I20260620 00:11:11.136966 22850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41a40a25b08f4c129ed0d73f40fc97de [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:11:11.139022 22802 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 41a40a25b08f4c129ed0d73f40fc97de [term 1 FOLLOWER]: Refusing update from remote peer 3d0e99784b0544c19502ed9b532443e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:11:11.139354 22854 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3d0e99784b0544c19502ed9b532443e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41a40a25b08f4c129ed0d73f40fc97de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 00:11:11.136677 22867 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 41a40a25b08f4c129ed0d73f40fc97de: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:11:11.141422 22867 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:11:11.138665 22760 master_runner.cc:186] Error getting master registration for 127.0.98.61:46017: OK, instance_id { permanent_uuid: "81ce1a59c0ab4663b2fce47847e4c429" instance_seqno: 1781914270918814 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 00:11:11.141857 22760 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 00:11:11.142163 22850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41a40a25b08f4c129ed0d73f40fc97de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d0e99784b0544c19502ed9b532443e9. Latest consensus state: current_term: 1 leader_uuid: "3d0e99784b0544c19502ed9b532443e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0e99784b0544c19502ed9b532443e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38007 } } peers { permanent_uuid: "81ce1a59c0ab4663b2fce47847e4c429" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46017 } } peers { permanent_uuid: "41a40a25b08f4c129ed0d73f40fc97de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37035 } } }
I20260620 00:11:11.142274 22850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41a40a25b08f4c129ed0d73f40fc97de [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:11:11.143671 22854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d0e99784b0544c19502ed9b532443e9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3d0e99784b0544c19502ed9b532443e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0e99784b0544c19502ed9b532443e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38007 } } peers { permanent_uuid: "81ce1a59c0ab4663b2fce47847e4c429" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46017 } } peers { permanent_uuid: "41a40a25b08f4c129ed0d73f40fc97de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37035 } } }
I20260620 00:11:11.143795 22854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d0e99784b0544c19502ed9b532443e9 [sys.catalog]: This master's current role is: LEADER
I20260620 00:11:11.144359 22850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41a40a25b08f4c129ed0d73f40fc97de [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3d0e99784b0544c19502ed9b532443e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0e99784b0544c19502ed9b532443e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38007 } } peers { permanent_uuid: "81ce1a59c0ab4663b2fce47847e4c429" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46017 } } peers { permanent_uuid: "41a40a25b08f4c129ed0d73f40fc97de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37035 } } }
I20260620 00:11:11.144447 22850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41a40a25b08f4c129ed0d73f40fc97de [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:11:11.144644 22870 mvcc.cc:205] Tried to move back new op lower bound from 7298720854581551104 to 7298720854569717760. Current Snapshot: MvccSnapshot[applied={T|T < 7298720854581551104}]
I20260620 00:11:11.144758 22869 mvcc.cc:205] Tried to move back new op lower bound from 7298720854581551104 to 7298720854569717760. Current Snapshot: MvccSnapshot[applied={T|T < 7298720854581551104}]
I20260620 00:11:11.152048 22757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:11.142009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41128 (local address 127.0.98.61:46017)
0620 00:11:11.142077 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:11.142079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:11.142621 (+ 542us) server_negotiation.cc:407] Connection header received
0620 00:11:11.142778 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:11.142779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:11.142791 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:11.142824 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:11.144616 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:11.145206 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:11.146407 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:11.146786 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:11.146871 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:11.148644 (+ 1773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:11.148645 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:11.148645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:11:11.148653 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:11.150498 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:11.151065 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:11.151066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:11.151067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:11.151139 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:11.151416 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:11.151417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:11.151417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:11.151567 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:11.151683 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:11.151852 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 00:11:11.151931 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 00:11:11.153744 22866 catalog_manager.cc:1347] Generated new cluster ID: f46604c114c1473b96f0ebea06395da8
I20260620 00:11:11.153808 22866 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:11:11.155237 22753 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37035 } attempt: 3
I20260620 00:11:11.158407 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:11.141761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41552 (local address 127.0.98.60:37035)
0620 00:11:11.141803 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:11.141805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:11.149542 (+ 7737us) server_negotiation.cc:407] Connection header received
0620 00:11:11.149675 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:11.149677 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:11.149692 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:11.149752 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:11.150408 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:11.151187 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:11.152846 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:11.153202 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:11.153265 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:11.154953 (+ 1688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:11.154955 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:11.154955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:11:11.154962 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:11.156900 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:11.157283 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:11.157284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:11.157285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:11.157357 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:11.157575 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:11.157575 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:11.157576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:11.157726 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:11.157846 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:11.158211 (+ 365us) server_negotiation.cc:299] Negotiation successful
0620 00:11:11.158300 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Jun 20 00:11:11 dist-test-slave-grqt krb5kdc[22597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 00:11:11.161284 22681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:11.142143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40110 (local address 127.0.98.62:38007)
0620 00:11:11.142203 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:11.142205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:11.149744 (+ 7539us) server_negotiation.cc:407] Connection header received
0620 00:11:11.150492 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:11.150494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:11.150506 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:11.150534 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:11.151135 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:11.151866 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:11.154641 (+ 2775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:11.154927 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:11.154963 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:11.157885 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:11.157886 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:11.157887 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:11:11.157894 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:11.159631 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:11.160515 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:11.160517 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:11.160517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:11.160568 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:11.160730 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:11.160730 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:11.160731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:11.160841 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:11.160932 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:11.161108 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 00:11:11.161193 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 00:11:11.164182 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:11.155486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37035 (local address 127.0.0.1:41562)
0620 00:11:11.155569 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:11.155580 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:11:11.155617 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:11:11.155787 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:11:11.155788 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:11:11.155939 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:11:11.155976 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:11:11.155977 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:11:11.156888 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:11:11.156890 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:11:11.157661 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:11:11.157663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:11:11.158160 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:11:11.158161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:11:11.158219 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:11.159717 (+ 1498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:11:11.159722 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:11:11.161158 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:11:11.163056 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:11:11.163057 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:11:11.163058 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:11:11.163288 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:11:11.163508 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:11:11.163509 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:11:11.163509 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:11:11.163596 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:11:11.163856 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:11:11.163857 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:11:11.163903 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 00:11:11.163980 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 00:11:11.164085 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 00:11:11.164183 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:11.155524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41562 (local address 127.0.98.60:37035)
0620 00:11:11.155677 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:11.155679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:11.155686 (+ 7us) server_negotiation.cc:407] Connection header received
0620 00:11:11.155704 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:11.155705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:11.155713 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:11.155753 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:11.156033 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:11.156806 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:11.157739 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:11.158041 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:11.158107 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:11.161245 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:11.161246 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:11.161246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:11:11.161253 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:11.162956 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:11.163354 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:11.163355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:11.163356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:11.163423 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:11.163658 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:11.163659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:11.163660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:11.163782 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:11.163850 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:11.164014 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 00:11:11.164094 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260620 00:11:11.165773 22753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81ce1a59c0ab4663b2fce47847e4c429: Bootstrap starting.
I20260620 00:11:11.166519 22753 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81ce1a59c0ab4663b2fce47847e4c429: Neither blocks nor log segments found. Creating new log.
I20260620 00:11:11.166827 22753 log.cc:824] T 00000000000000000000000000000000 P 81ce1a59c0ab4663b2fce47847e4c429: Log is configured to *not* fsync() on all Append() calls
I20260620 00:11:11.167686 22753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81ce1a59c0ab4663b2fce47847e4c429: No bootstrap required, opened a new log
I20260620 00:11:11.169839 22753 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81ce1a59c0ab4663b2fce47847e4c429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0e99784b0544c19502ed9b532443e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38007 } } peers { permanent_uuid: "81ce1a59c0ab4663b2fce47847e4c429" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46017 } } peers { permanent_uuid: "41a40a25b08f4c129ed0d73f40fc97de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37035 } }
I20260620 00:11:11.169987 22753 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81ce1a59c0ab4663b2fce47847e4c429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81ce1a59c0ab4663b2fce47847e4c429, State: Initialized, Role: FOLLOWER
I20260620 00:11:11.170073 22753 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81ce1a59c0ab4663b2fce47847e4c429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0e99784b0544c19502ed9b532443e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38007 } } peers { permanent_uuid: "81ce1a59c0ab4663b2fce47847e4c429" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46017 } } peers { permanent_uuid: "41a40a25b08f4c129ed0d73f40fc97de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37035 } }
I20260620 00:11:11.170517 22726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81ce1a59c0ab4663b2fce47847e4c429 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:11:11.170405 22878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81ce1a59c0ab4663b2fce47847e4c429 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0e99784b0544c19502ed9b532443e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38007 } } peers { permanent_uuid: "81ce1a59c0ab4663b2fce47847e4c429" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46017 } } peers { permanent_uuid: "41a40a25b08f4c129ed0d73f40fc97de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37035 } } }
I20260620 00:11:11.170712 22878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81ce1a59c0ab4663b2fce47847e4c429 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:11:11.171314 22726 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 81ce1a59c0ab4663b2fce47847e4c429 [term 1 FOLLOWER]: Refusing update from remote peer 3d0e99784b0544c19502ed9b532443e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:11:11.170526 22727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3d0e99784b0544c19502ed9b532443e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "81ce1a59c0ab4663b2fce47847e4c429"
I20260620 00:11:11.171948 22836 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3d0e99784b0544c19502ed9b532443e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81ce1a59c0ab4663b2fce47847e4c429" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:11:11.172610 22727 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 81ce1a59c0ab4663b2fce47847e4c429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d0e99784b0544c19502ed9b532443e9 in term 1.
I20260620 00:11:11.170469 22753 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81ce1a59c0ab4663b2fce47847e4c429 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:11:11.174008 22878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81ce1a59c0ab4663b2fce47847e4c429 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d0e99784b0544c19502ed9b532443e9. Latest consensus state: current_term: 1 leader_uuid: "3d0e99784b0544c19502ed9b532443e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0e99784b0544c19502ed9b532443e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38007 } } peers { permanent_uuid: "81ce1a59c0ab4663b2fce47847e4c429" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46017 } } peers { permanent_uuid: "41a40a25b08f4c129ed0d73f40fc97de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37035 } } }
I20260620 00:11:11.174439 22878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81ce1a59c0ab4663b2fce47847e4c429 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:11:11.175470 22879 mvcc.cc:205] Tried to move back new op lower bound from 7298720854581551104 to 7298720854569717760. Current Snapshot: MvccSnapshot[applied={T|T < 7298720854581551104}]
I20260620 00:11:11.175590 22878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81ce1a59c0ab4663b2fce47847e4c429 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3d0e99784b0544c19502ed9b532443e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0e99784b0544c19502ed9b532443e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38007 } } peers { permanent_uuid: "81ce1a59c0ab4663b2fce47847e4c429" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46017 } } peers { permanent_uuid: "41a40a25b08f4c129ed0d73f40fc97de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37035 } } }
I20260620 00:11:11.175658 22878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81ce1a59c0ab4663b2fce47847e4c429 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 00:11:11.176378 22893 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 81ce1a59c0ab4663b2fce47847e4c429: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:11:11.176482 22893 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:11:11.190207 22836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d0e99784b0544c19502ed9b532443e9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3d0e99784b0544c19502ed9b532443e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0e99784b0544c19502ed9b532443e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38007 } } peers { permanent_uuid: "81ce1a59c0ab4663b2fce47847e4c429" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46017 } } peers { permanent_uuid: "41a40a25b08f4c129ed0d73f40fc97de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37035 } } }
I20260620 00:11:11.190333 22836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d0e99784b0544c19502ed9b532443e9 [sys.catalog]: This master's current role is: LEADER
I20260620 00:11:11.192934 22866 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:11:11.193754 22866 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:11:11.199712 22866 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d0e99784b0544c19502ed9b532443e9: Generated new TSK 0
I20260620 00:11:11.199923 22866 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 00:11:11.227141 22681 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:11:11.225532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40116 (local address 127.0.98.62:38007)
0620 00:11:11.225577 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:11.225580 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:11.226452 (+ 872us) server_negotiation.cc:407] Connection header received
0620 00:11:11.226562 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:11.226563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:11.226574 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:11.226606 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:11.227067 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40116: BlockingRecv error: recv got EOF from 127.0.0.1:40116 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 00:11:11.227922 22757 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:11:11.225954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41144 (local address 127.0.98.61:46017)
0620 00:11:11.225995 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:11.225997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:11.227400 (+ 1403us) server_negotiation.cc:407] Connection header received
0620 00:11:11.227501 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:11.227502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:11.227513 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:11.227542 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:11.227861 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41144: BlockingRecv error: recv got EOF from 127.0.0.1:41144 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 00:11:11.228639 22832 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:11:11.226308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41578 (local address 127.0.98.60:37035)
0620 00:11:11.226350 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:11.226352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:11.228142 (+ 1790us) server_negotiation.cc:407] Connection header received
0620 00:11:11.228237 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:11.228238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:11.228249 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:11.228279 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:11.228575 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41578: BlockingRecv error: recv got EOF from 127.0.0.1:41578 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 00:11:11.231067 22757 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:11:11.230293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41150 (local address 127.0.98.61:46017)
0620 00:11:11.230327 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:11.230329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:11.230541 (+ 212us) server_negotiation.cc:407] Connection header received
0620 00:11:11.230647 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:11.230648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:11.230667 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:11.230710 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:11.231020 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41150: BlockingRecv error: recv got EOF from 127.0.0.1:41150 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 00:11:11.234090 22757 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:11:11.233320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41154 (local address 127.0.98.61:46017)
0620 00:11:11.233353 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:11.233355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:11.233613 (+ 258us) server_negotiation.cc:407] Connection header received
0620 00:11:11.233721 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:11.233722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:11.233732 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:11.233760 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:11.234045 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41154: BlockingRecv error: recv got EOF from 127.0.0.1:41154 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 00:11:11.237928 22757 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:11:11.237195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41170 (local address 127.0.98.61:46017)
0620 00:11:11.237227 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:11.237229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:11.237466 (+ 237us) server_negotiation.cc:407] Connection header received
0620 00:11:11.237553 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:11.237554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:11.237563 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:11.237590 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:11.237888 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41170: BlockingRecv error: recv got EOF from 127.0.0.1:41170 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 00:11:11.244161 22757 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:11:11.243322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41172 (local address 127.0.98.61:46017)
0620 00:11:11.243360 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:11.243362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:11.243602 (+ 240us) server_negotiation.cc:407] Connection header received
0620 00:11:11.243720 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:11.243721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:11.243733 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:11.243763 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:11.244110 (+ 347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41172: BlockingRecv error: recv got EOF from 127.0.0.1:41172 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 00:11:11.254580 22757 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:11:11.253619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41180 (local address 127.0.98.61:46017)
0620 00:11:11.253664 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:11.253666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:11.253962 (+ 296us) server_negotiation.cc:407] Connection header received
0620 00:11:11.254072 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:11.254073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:11.254085 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:11.254115 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:11.254515 (+ 400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41180: BlockingRecv error: recv got EOF from 127.0.0.1:41180 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 00:11:11.272938 22757 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:11:11.271995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41186 (local address 127.0.98.61:46017)
0620 00:11:11.272040 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:11.272042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:11.272319 (+ 277us) server_negotiation.cc:407] Connection header received
0620 00:11:11.272428 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:11.272429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:11.272441 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:11.272472 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:11.272878 (+ 406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41186: BlockingRecv error: recv got EOF from 127.0.0.1:41186 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 00:11:11.307354 22757 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:11:11.306417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41192 (local address 127.0.98.61:46017)
0620 00:11:11.306462 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:11.306464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:11.306755 (+ 291us) server_negotiation.cc:407] Connection header received
0620 00:11:11.306863 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:11.306864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:11.306875 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:11.306906 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:11.307291 (+ 385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41192: BlockingRecv error: recv got EOF from 127.0.0.1:41192 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 00:11:11.373579 22757 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:11:11.372808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41202 (local address 127.0.98.61:46017)
0620 00:11:11.372886 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:11.372888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:11.372955 (+ 67us) server_negotiation.cc:407] Connection header received
0620 00:11:11.373007 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:11.373008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:11.373018 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:11.373090 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:11.373508 (+ 418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41202: BlockingRecv error: recv got EOF from 127.0.0.1:41202 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 00:11:11.503496 22757 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:11:11.502798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41210 (local address 127.0.98.61:46017)
0620 00:11:11.502875 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:11.502878 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:11.502970 (+ 92us) server_negotiation.cc:407] Connection header received
0620 00:11:11.503023 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:11.503024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:11.503043 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:11.503112 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:11.503426 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41210: BlockingRecv error: recv got EOF from 127.0.0.1:41210 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260620 00:11:11.761353 22757 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:11:11.760731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41212 (local address 127.0.98.61:46017)
0620 00:11:11.760813 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:11.760816 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:11.760833 (+ 17us) server_negotiation.cc:407] Connection header received
0620 00:11:11.760862 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:11.760862 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:11.760872 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:11.760935 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:11.761269 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41212: BlockingRecv error: recv got EOF from 127.0.0.1:41212 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 00:11:12.141997 22867 catalog_manager.cc:1259] Loaded cluster ID: f46604c114c1473b96f0ebea06395da8
I20260620 00:11:12.142071 22867 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 41a40a25b08f4c129ed0d73f40fc97de: loading cluster ID for follower catalog manager: success
I20260620 00:11:12.142894 22867 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 41a40a25b08f4c129ed0d73f40fc97de: acquiring CA information for follower catalog manager: success
I20260620 00:11:12.143245 22867 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 41a40a25b08f4c129ed0d73f40fc97de: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 00:11:12.176930 22893 catalog_manager.cc:1259] Loaded cluster ID: f46604c114c1473b96f0ebea06395da8
I20260620 00:11:12.177001 22893 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 81ce1a59c0ab4663b2fce47847e4c429: loading cluster ID for follower catalog manager: success
I20260620 00:11:12.177870 22893 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 81ce1a59c0ab4663b2fce47847e4c429: acquiring CA information for follower catalog manager: success
I20260620 00:11:12.178153 22893 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 81ce1a59c0ab4663b2fce47847e4c429: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 00:11:12.277251 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:12.274571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41224 (local address 127.0.98.61:46017)
0620 00:11:12.274705 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:12.274710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:12.274856 (+ 146us) server_negotiation.cc:407] Connection header received
0620 00:11:12.274968 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:12.274969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:12.275026 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:12.275109 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:11:12.275262 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:12.275777 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:12.276559 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:12.276754 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:12.276808 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:12.276945 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:11:12.277054 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:11:12.277113 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:12.277165 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 00:11:12.277183 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":53,"threads_started":1}
I20260620 00:11:12.281359 22966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:12.279087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41582 (local address 127.0.98.60:37035)
0620 00:11:12.279197 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:12.279200 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:12.279208 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:11:12.279255 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:12.279256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:12.279289 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:12.279359 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:11:12.279458 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:12.279967 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:12.280676 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:12.280884 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:12.280948 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:12.281070 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:11:12.281168 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:11:12.281226 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:12.281269 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 00:11:12.281287 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":69,"threads_started":1}
I20260620 00:11:12.285775 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:12.282990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41230 (local address 127.0.98.61:46017)
0620 00:11:12.283033 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:12.283035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:12.283244 (+ 209us) server_negotiation.cc:407] Connection header received
0620 00:11:12.283296 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:12.283297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:12.283305 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:12.283378 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:11:12.283498 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:12.284253 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:12.285043 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:12.285250 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:12.285334 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:12.285514 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:11:12.285545 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:11:12.285597 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:12.285661 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 00:11:12.285682 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 00:11:12.286374 22966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:12.283096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41588 (local address 127.0.98.60:37035)
0620 00:11:12.283139 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:12.283140 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:12.283490 (+ 350us) server_negotiation.cc:407] Connection header received
0620 00:11:12.283579 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:12.283580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:12.283588 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:12.283642 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:11:12.283756 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:12.284436 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:12.285445 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:12.285719 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:12.285775 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:12.286072 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:11:12.286102 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:11:12.286154 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:12.286275 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 00:11:12.286300 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 00:11:12.286481 22971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:12.282997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40120 (local address 127.0.98.62:38007)
0620 00:11:12.283184 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:12.283186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:12.283361 (+ 175us) server_negotiation.cc:407] Connection header received
0620 00:11:12.283437 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:12.283438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:12.283491 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:12.283571 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:11:12.283800 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:12.284291 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:12.285414 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:12.285748 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:12.285808 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:12.286052 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:11:12.286187 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:11:12.286244 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:12.286362 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 00:11:12.286384 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":49,"threads_started":1}
I20260620 00:11:12.288183 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 22610
I20260620 00:11:12.293247 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 22685
I20260620 00:11:12.298394 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 22760
2026-06-20T00:11:12Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2808 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:11:12 dist-test-slave-grqt krb5kdc[22977](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:11:12 dist-test-slave-grqt krb5kdc[22977](info): set up 2 sockets
Jun 20 00:11:12 dist-test-slave-grqt krb5kdc[22977](info): commencing operation
krb5kdc: starting...
W20260620 00:11:13.329701 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/krb5kdc/test-user.keytab.
Jun 20 00:11:13 dist-test-slave-grqt krb5kdc[22977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:11:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:11:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:11:13.448238 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32833
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32833,127.0.98.61:40037,127.0.98.60:43557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:11:13.531724 22990 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 00:11:13.531976 22990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:11:13.532008 22990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:11:13.532055 22990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:11:13.533186 22990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:11:13.533254 22990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:11:13.533279 22990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:11:13.533305 22990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:11:13.533336 22990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:11:13.534832 22990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32833,127.0.98.61:40037,127.0.98.60:43557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:11:13.535110 22990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:11:13.535341 22990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:11:13.540324 22996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:11:13.540314 22995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:11:13.540300 22998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:11:13.540625 22990 server_base.cc:1034] running on GCE node
I20260620 00:11:13.540871 22990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:11:13.541114 22990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:11:13.542256 22990 hybrid_clock.cc:648] HybridClock initialized: now 1781914273542244 us; error 30 us; skew 500 ppm
Jun 20 00:11:13 dist-test-slave-grqt krb5kdc[22977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:11:13.544745 22990 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:11:13.545234 22990 webserver.cc:460] Webserver started at http://127.0.98.62:42815/ using document root <none> and password file <none>
I20260620 00:11:13.545506 22990 fs_manager.cc:359] Metadata directory not provided
I20260620 00:11:13.545555 22990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:11:13.545713 22990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:11:13.546782 22990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-0/data/instance:
uuid: "892e00cdd723432c9a76fe82dd40112f"
format_stamp: "Formatted at 2026-06-20 00:11:13 on dist-test-slave-grqt"
server_key: "05a1ae6881808ceae4bbfd9ddcd45773"
server_key_iv: "c5ad594d89fdc59bbd4dc6a6d2ee951d"
server_key_version: "encryptionkey@0"
I20260620 00:11:13.547144 22990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-0/wal/instance:
uuid: "892e00cdd723432c9a76fe82dd40112f"
format_stamp: "Formatted at 2026-06-20 00:11:13 on dist-test-slave-grqt"
server_key: "05a1ae6881808ceae4bbfd9ddcd45773"
server_key_iv: "c5ad594d89fdc59bbd4dc6a6d2ee951d"
server_key_version: "encryptionkey@0"
I20260620 00:11:13.548460 22990 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:11:13.549216 23005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:11:13.549435 22990 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:11:13.549525 22990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-0/wal
uuid: "892e00cdd723432c9a76fe82dd40112f"
format_stamp: "Formatted at 2026-06-20 00:11:13 on dist-test-slave-grqt"
server_key: "05a1ae6881808ceae4bbfd9ddcd45773"
server_key_iv: "c5ad594d89fdc59bbd4dc6a6d2ee951d"
server_key_version: "encryptionkey@0"
I20260620 00:11:13.549640 22990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:11:13.562417 22990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:11:13.564272 22990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:11:13.564435 22990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:11:13.568360 22990 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32833
I20260620 00:11:13.568394 23057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32833 every 8 connection(s)
I20260620 00:11:13.568745 22990 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:11:13.569773 23058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:11:13.571262 23058 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32833 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:11:13.573167 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 22990
I20260620 00:11:13.573279 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-0/wal/instance
I20260620 00:11:13.573498 392 external_mini_cluster.cc:1428] Setting key 2f8b8442abaaa6c0ce91d7b7f6fe7d59
Jun 20 00:11:13 dist-test-slave-grqt krb5kdc[22977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260620 00:11:13.582060 23061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:13.571921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54016 (local address 127.0.98.62:32833)
0620 00:11:13.572078 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:13.572082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:13.572141 (+ 59us) server_negotiation.cc:407] Connection header received
0620 00:11:13.572646 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:13.572647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:13.572664 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:13.572747 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:13.573302 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.574630 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.575976 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.576293 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.576375 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:13.578807 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:13.578815 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:13.578815 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:11:13.578827 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:13.580354 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:13.580760 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:13.580763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:13.580763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:13.580844 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:13.581079 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:13.581080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:13.581080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:13.581324 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:13.581463 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:13.581665 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 00:11:13.581813 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
I20260620 00:11:13.582058 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:13.571858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32833 (local address 127.0.0.1:54016)
0620 00:11:13.572013 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:13.572056 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 00:11:13.572513 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:11:13.572775 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:11:13.572777 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:11:13.573019 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:11:13.573174 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:11:13.573178 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.574737 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.574738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:11:13.575871 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:11:13.575874 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.576406 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.576407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:11:13.576492 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:13.577011 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:11:13.577021 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:11:13.578702 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:11:13.580451 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:11:13.580456 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:11:13.580459 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:11:13.580663 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:11:13.580911 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:11:13.580912 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:11:13.580913 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:11:13.581003 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:11:13.581428 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:11:13.581433 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:11:13.581513 (+ 80us) client_negotiation.cc:769] Sending connection context
0620 00:11:13.581633 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 00:11:13.581878 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":100,"thread_start_us":84,"threads_started":1}
I20260620 00:11:13.582916 23058 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40037 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:11:13.583323 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:13.583154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40037 (local address 127.0.0.1:33556)
0620 00:11:13.583216 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:13.583274 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 00:11:13.583498 23006 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40037: connect: Connection refused (error 111)
I20260620 00:11:13.583914 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:13.583803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40037 (local address 127.0.0.1:33558)
0620 00:11:13.583868 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:13.583897 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
W20260620 00:11:13.584053 23058 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40037: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40037: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 00:11:13.600239 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40037
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32833,127.0.98.61:40037,127.0.98.60:43557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:11:13.633318 23058 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40037 } attempt: 1
I20260620 00:11:13.633916 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:13.633776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40037 (local address 127.0.0.1:33564)
0620 00:11:13.633844 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:13.633886 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 00:11:13.634238 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:13.634143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40037 (local address 127.0.0.1:33570)
0620 00:11:13.634188 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:13.634212 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 00:11:13.634377 23058 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40037: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40037: connect: Connection refused (error 111)
W20260620 00:11:13.682935 23065 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 00:11:13.683161 23065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:11:13.683179 23065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:11:13.683200 23065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:11:13.684398 23065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:11:13.684444 23065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:11:13.684459 23065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:11:13.684471 23065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:11:13.684525 23065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:11:13.686048 23065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32833,127.0.98.61:40037,127.0.98.60:43557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:11:13.686322 23065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:11:13.686563 23065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:11:13.691449 23071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:11:13.691455 23070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:11:13.691473 23073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:11:13.691686 23065 server_base.cc:1034] running on GCE node
I20260620 00:11:13.691978 23065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:11:13.692310 23065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:11:13.693436 23065 hybrid_clock.cc:648] HybridClock initialized: now 1781914273693411 us; error 38 us; skew 500 ppm
Jun 20 00:11:13 dist-test-slave-grqt krb5kdc[22977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:11:13.695868 23065 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 00:11:13.696368 23065 webserver.cc:460] Webserver started at http://127.0.98.61:35763/ using document root <none> and password file <none>
I20260620 00:11:13.696616 23065 fs_manager.cc:359] Metadata directory not provided
I20260620 00:11:13.696689 23065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:11:13.696836 23065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:11:13.697989 23065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-1/data/instance:
uuid: "997f0e95727940b19036ed7058c27008"
format_stamp: "Formatted at 2026-06-20 00:11:13 on dist-test-slave-grqt"
server_key: "83d0afc064fc7d5c4b7b7e84329e1be8"
server_key_iv: "cc3133014c90d00a6652542c71c60b96"
server_key_version: "encryptionkey@0"
I20260620 00:11:13.698364 23065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-1/wal/instance:
uuid: "997f0e95727940b19036ed7058c27008"
format_stamp: "Formatted at 2026-06-20 00:11:13 on dist-test-slave-grqt"
server_key: "83d0afc064fc7d5c4b7b7e84329e1be8"
server_key_iv: "cc3133014c90d00a6652542c71c60b96"
server_key_version: "encryptionkey@0"
I20260620 00:11:13.699747 23065 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:11:13.700584 23080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:11:13.700778 23065 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:11:13.700891 23065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-1/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-1/wal
uuid: "997f0e95727940b19036ed7058c27008"
format_stamp: "Formatted at 2026-06-20 00:11:13 on dist-test-slave-grqt"
server_key: "83d0afc064fc7d5c4b7b7e84329e1be8"
server_key_iv: "cc3133014c90d00a6652542c71c60b96"
server_key_version: "encryptionkey@0"
I20260620 00:11:13.700970 23065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:11:13.702528 23058 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40037 } attempt: 2
I20260620 00:11:13.702922 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:13.702801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40037 (local address 127.0.0.1:33578)
0620 00:11:13.702857 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:13.702895 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 00:11:13.703256 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:13.703162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40037 (local address 127.0.0.1:33592)
0620 00:11:13.703204 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:13.703228 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 00:11:13.703338 23058 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40037: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40037: connect: Connection refused (error 111)
I20260620 00:11:13.718992 23065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:11:13.720863 23065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:11:13.721037 23065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:11:13.725313 23065 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40037
I20260620 00:11:13.725409 23132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40037 every 8 connection(s)
I20260620 00:11:13.725778 23065 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-1/data/info.pb
I20260620 00:11:13.726783 23133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:11:13.728407 23133 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32833 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 00:11:13 dist-test-slave-grqt krb5kdc[22977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:11:13.735440 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 23065
I20260620 00:11:13.735541 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-1/wal/instance
I20260620 00:11:13.735718 392 external_mini_cluster.cc:1428] Setting key a9fa85ea4ed65776615154ae18b431c2
I20260620 00:11:13.738662 23061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:13.729202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54024 (local address 127.0.98.62:32833)
0620 00:11:13.729303 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:13.729305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:13.729537 (+ 232us) server_negotiation.cc:407] Connection header received
0620 00:11:13.730035 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:13.730036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:13.730050 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:13.730114 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:13.730781 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.731310 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.732325 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.732586 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.732655 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:13.735008 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:13.735009 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:13.735009 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:11:13.735017 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:13.737010 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:13.737626 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:13.737627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:13.737628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:13.737706 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:13.737958 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:13.737959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:13.737959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:13.738106 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:13.738160 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:13.738425 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 00:11:13.738521 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 00:11:13.738730 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:13.729146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32833 (local address 127.0.0.1:54024)
0620 00:11:13.729404 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:13.729451 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 00:11:13.729943 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:11:13.730189 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:11:13.730193 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:11:13.730446 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:11:13.730665 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:11:13.730670 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.731407 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.731407 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:11:13.732235 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:11:13.732237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.732682 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.732682 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:11:13.732790 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:13.733301 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:11:13.733316 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:11:13.734900 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:11:13.737119 (+ 2219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:11:13.737126 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:11:13.737128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:11:13.737548 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:11:13.737777 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:11:13.737778 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:11:13.737778 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:11:13.737885 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:11:13.738161 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:11:13.738165 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:11:13.738287 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 00:11:13.738375 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 00:11:13.738501 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":134,"threads_started":1}
I20260620 00:11:13.739326 23133 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40037 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Jun 20 00:11:13 dist-test-slave-grqt krb5kdc[22977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260620 00:11:13.747025 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:13.739642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40037 (local address 127.0.0.1:33602)
0620 00:11:13.739674 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:13.739685 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:11:13.739727 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:11:13.739995 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:11:13.739996 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:11:13.740141 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:11:13.740187 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:11:13.740188 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.741034 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.741035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:11:13.741630 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:11:13.741632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.742138 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.742139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:11:13.742208 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:13.742698 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:11:13.742703 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:11:13.743898 (+ 1195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:11:13.745890 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:11:13.745905 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:11:13.745906 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:11:13.746062 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:11:13.746287 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:11:13.746288 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:11:13.746289 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:11:13.746347 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:11:13.746663 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:11:13.746664 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:11:13.746704 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 00:11:13.746791 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 00:11:13.746899 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260620 00:11:13.747102 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:13.739718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33602 (local address 127.0.98.61:40037)
0620 00:11:13.739856 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:13.739861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:13.739878 (+ 17us) server_negotiation.cc:407] Connection header received
0620 00:11:13.739897 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:13.739899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:13.739917 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:13.739994 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:13.740261 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.740935 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.741720 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.742048 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.742138 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:13.743983 (+ 1845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:13.743992 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:13.743993 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:11:13.743999 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:13.745792 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:13.746148 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:13.746151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:13.746152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:13.746215 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:13.746431 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:13.746431 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:13.746432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:13.746584 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:13.746674 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:13.746822 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 00:11:13.746973 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
I20260620 00:11:13.747730 23133 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:11:13.748132 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:13.748015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43557 (local address 127.0.0.1:39012)
0620 00:11:13.748063 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:13.748109 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 00:11:13.748293 23084 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43557: connect: Connection refused (error 111)
I20260620 00:11:13.748731 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:13.748644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43557 (local address 127.0.0.1:39018)
0620 00:11:13.748690 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:13.748715 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 00:11:13.748857 23133 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43557: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43557: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 00:11:13.762487 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8WXHPi/build/release/bin/kudu
/tmp/dist-test-task8WXHPi/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43557
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32833,127.0.98.61:40037,127.0.98.60:43557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:11:13.794540 23058 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40037 } attempt: 3
I20260620 00:11:13.798105 23133 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } attempt: 1
I20260620 00:11:13.798532 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:13.798375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43557 (local address 127.0.0.1:39020)
0620 00:11:13.798447 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:13.798495 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
Jun 20 00:11:13 dist-test-slave-grqt krb5kdc[22977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 00:11:13.798938 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:13.798829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43557 (local address 127.0.0.1:39036)
0620 00:11:13.798880 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:13.798909 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 00:11:13.799031 23133 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43557: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43557: connect: Connection refused (error 111)
I20260620 00:11:13.802343 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:13.794972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40037 (local address 127.0.0.1:33616)
0620 00:11:13.795037 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:13.795047 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:11:13.795107 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:11:13.795265 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:11:13.795266 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:11:13.795432 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:11:13.795496 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:11:13.795498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.796182 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.796183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:11:13.796704 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:11:13.796706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.797112 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.797113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:11:13.797195 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:13.797757 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:11:13.797762 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:11:13.799251 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:11:13.801199 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:11:13.801201 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:11:13.801201 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:11:13.801426 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:11:13.801683 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:11:13.801684 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:11:13.801684 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:11:13.801747 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:11:13.802051 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:11:13.802052 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:11:13.802100 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 00:11:13.802169 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 00:11:13.802263 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 00:11:13.802389 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:13.795042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33616 (local address 127.0.98.61:40037)
0620 00:11:13.795112 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:13.795114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:13.795146 (+ 32us) server_negotiation.cc:407] Connection header received
0620 00:11:13.795185 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:13.795186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:13.795197 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:13.795260 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:13.795580 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.796082 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.796790 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.797021 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.797111 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:13.799338 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:13.799340 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:13.799340 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:11:13.799348 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:13.801082 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:13.801518 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:13.801520 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:13.801520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:13.801590 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:13.801817 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:13.801818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:13.801818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:13.801965 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:13.802038 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:13.802213 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 00:11:13.802300 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 00:11:13.802933 23058 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:11:13.803300 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:13.803184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43557 (local address 127.0.0.1:39042)
0620 00:11:13.803238 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:13.803274 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 00:11:13.803620 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:13.803534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43557 (local address 127.0.0.1:39058)
0620 00:11:13.803575 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:13.803601 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 00:11:13.803712 23058 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43557: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43557: connect: Connection refused (error 111)
I20260620 00:11:13.834872 23058 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } attempt: 1
I20260620 00:11:13.835444 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:13.835268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43557 (local address 127.0.0.1:39066)
0620 00:11:13.835340 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:13.835399 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 00:11:13.835903 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:13.835785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43557 (local address 127.0.0.1:39074)
0620 00:11:13.835848 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:13.835877 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 00:11:13.836050 23058 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43557: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43557: connect: Connection refused (error 111)
W20260620 00:11:13.856738 23140 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 00:11:13.857033 23140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:11:13.857076 23140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:11:13.857111 23140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:11:13.858480 23140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:11:13.858536 23140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:11:13.858552 23140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:11:13.858566 23140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:11:13.858603 23140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:11:13.860378 23140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32833,127.0.98.61:40037,127.0.98.60:43557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:11:13.860694 23140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:11:13.860980 23140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 00:11:13.866343 23140 server_base.cc:1034] running on GCE node
W20260620 00:11:13.866313 23148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:11:13.866310 23145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:11:13.866345 23146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:11:13.866865 23140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:11:13.867182 23140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:11:13.867165 23133 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } attempt: 2
I20260620 00:11:13.867550 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:13.867427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43557 (local address 127.0.0.1:39084)
0620 00:11:13.867486 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:13.867520 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260620 00:11:13.867838 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:13.867784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43557 (local address 127.0.0.1:39100)
0620 00:11:13.867804 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:13.867825 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260620 00:11:13.867923 23133 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43557: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43557: connect: Connection refused (error 111)
I20260620 00:11:13.868351 23140 hybrid_clock.cc:648] HybridClock initialized: now 1781914273868332 us; error 39 us; skew 500 ppm
Jun 20 00:11:13 dist-test-slave-grqt krb5kdc[22977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:11:13.871073 23140 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 00:11:13.871547 23140 webserver.cc:460] Webserver started at http://127.0.98.60:46301/ using document root <none> and password file <none>
I20260620 00:11:13.871807 23140 fs_manager.cc:359] Metadata directory not provided
I20260620 00:11:13.871855 23140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:11:13.872020 23140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:11:13.873097 23140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-2/data/instance:
uuid: "e884fa7572614f6eb70b2938a986c753"
format_stamp: "Formatted at 2026-06-20 00:11:13 on dist-test-slave-grqt"
server_key: "229a5cde5df097b882dc54c4a1666754"
server_key_iv: "597dd9b14e6c8100fce6d5b526b7117d"
server_key_version: "encryptionkey@0"
I20260620 00:11:13.873505 23140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-2/wal/instance:
uuid: "e884fa7572614f6eb70b2938a986c753"
format_stamp: "Formatted at 2026-06-20 00:11:13 on dist-test-slave-grqt"
server_key: "229a5cde5df097b882dc54c4a1666754"
server_key_iv: "597dd9b14e6c8100fce6d5b526b7117d"
server_key_version: "encryptionkey@0"
I20260620 00:11:13.874915 23140 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:11:13.875735 23155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:11:13.875890 23140 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:11:13.875984 23140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-2/data,/tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-2/wal
uuid: "e884fa7572614f6eb70b2938a986c753"
format_stamp: "Formatted at 2026-06-20 00:11:13 on dist-test-slave-grqt"
server_key: "229a5cde5df097b882dc54c4a1666754"
server_key_iv: "597dd9b14e6c8100fce6d5b526b7117d"
server_key_version: "encryptionkey@0"
I20260620 00:11:13.876042 23140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:11:13.891481 23140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:11:13.893287 23140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:11:13.893468 23140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:11:13.897154 23140 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43557
I20260620 00:11:13.897212 23207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43557 every 8 connection(s)
I20260620 00:11:13.897593 23140 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-2/data/info.pb
I20260620 00:11:13.897866 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8WXHPi/build/release/bin/kudu as pid 23140
I20260620 00:11:13.897969 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8WXHPi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914001714546-392-0/minicluster-data/master-2/wal/instance
I20260620 00:11:13.898212 392 external_mini_cluster.cc:1428] Setting key 08b076f477dabd92a8f67eee8b4c4d7e
I20260620 00:11:13.898885 23208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:11:13.900928 23208 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32833 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 00:11:13 dist-test-slave-grqt krb5kdc[22977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 00:11:13 dist-test-slave-grqt krb5kdc[22977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:11:13.909345 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:13.898782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39102 (local address 127.0.98.60:43557)
0620 00:11:13.898964 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:13.898974 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:13.899003 (+ 29us) server_negotiation.cc:407] Connection header received
0620 00:11:13.899561 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:13.899570 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:13.899647 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:13.899797 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:13.900449 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.901841 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.902872 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.903454 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.903532 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:13.906243 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:13.906250 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:13.906251 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:11:13.906258 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:13.907809 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:13.908194 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:13.908197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:13.908198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:13.908290 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:13.908498 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:13.908499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:13.908500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:13.908750 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:13.908904 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:13.909014 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 00:11:13.909143 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":62,"threads_started":1}
I20260620 00:11:13.909650 23061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:13.901696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54032 (local address 127.0.98.62:32833)
0620 00:11:13.901758 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:13.901761 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:13.901896 (+ 135us) server_negotiation.cc:407] Connection header received
0620 00:11:13.901968 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:13.901969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:13.901980 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:13.902033 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:13.902597 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.903203 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.904285 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.904468 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.904517 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:13.906857 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:13.906858 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:13.906858 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:11:13.906865 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:13.908331 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:13.908762 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:13.908764 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:13.908764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:13.908827 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:13.909025 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:13.909026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:13.909026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:13.909146 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:13.909230 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:13.909480 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 00:11:13.909560 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 00:11:13.909674 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:13.901608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32833 (local address 127.0.0.1:54032)
0620 00:11:13.901789 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:13.901827 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 00:11:13.901912 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:11:13.902069 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:11:13.902072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:11:13.902432 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:11:13.902523 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:11:13.902525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.903304 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.903305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:11:13.904188 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:11:13.904191 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.904572 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.904573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:11:13.904675 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:13.905087 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:11:13.905097 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:11:13.906764 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:11:13.908442 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:11:13.908448 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:11:13.908450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:11:13.908653 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:11:13.908896 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:11:13.908897 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:11:13.908897 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:11:13.908959 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:11:13.909259 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:11:13.909266 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:11:13.909319 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 00:11:13.909450 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 00:11:13.909560 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":105,"threads_started":1}
I20260620 00:11:13.910315 23208 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40037 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:11:13.911219 23058 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } attempt: 2
Jun 20 00:11:13 dist-test-slave-grqt krb5kdc[22977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 00:11:13 dist-test-slave-grqt krb5kdc[22977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 00:11:13 dist-test-slave-grqt krb5kdc[22977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 00:11:13.917881 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:13.910653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33632 (local address 127.0.98.61:40037)
0620 00:11:13.910698 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:13.910700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:13.910707 (+ 7us) server_negotiation.cc:407] Connection header received
0620 00:11:13.910731 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:13.910732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:13.910740 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:13.910796 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:13.911059 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.911738 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.912707 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.913002 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.913043 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:13.915035 (+ 1992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:13.915036 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:13.915036 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:11:13.915044 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:13.916732 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:13.917057 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:13.917058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:13.917059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:13.917118 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:13.917328 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:13.917329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:13.917329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:13.917494 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:13.917557 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:13.917716 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 00:11:13.917789 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 00:11:13.917881 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:13.910567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40037 (local address 127.0.0.1:33632)
0620 00:11:13.910621 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:13.910633 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:11:13.910671 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:11:13.910800 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:11:13.910801 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:11:13.910936 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:11:13.910982 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:11:13.910983 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.911829 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.911830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:11:13.912595 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:11:13.912596 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.913122 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.913123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:11:13.913179 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:13.913577 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:11:13.913582 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:11:13.914950 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:11:13.916821 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:11:13.916822 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:11:13.916822 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:11:13.916984 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:11:13.917189 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:11:13.917195 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:11:13.917195 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:11:13.917247 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:11:13.917548 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:11:13.917549 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:11:13.917593 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 00:11:13.917679 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 00:11:13.917779 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 00:11:13.918498 23208 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:11:13.919637 23061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:13.910232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54046 (local address 127.0.98.62:32833)
0620 00:11:13.910267 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:13.910269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:13.910285 (+ 16us) server_negotiation.cc:407] Connection header received
0620 00:11:13.910346 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:13.910346 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:13.910354 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:13.910394 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:13.910926 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.911616 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.912268 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.912471 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.912554 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:13.915714 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:13.915715 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:13.915716 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:11:13.915723 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:13.917357 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:13.918360 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:13.918362 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:13.918362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:13.918423 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:13.918659 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:13.918660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:13.918660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:13.918788 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:13.918959 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:13.919415 (+ 456us) server_negotiation.cc:299] Negotiation successful
0620 00:11:13.919512 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 00:11:13.920248 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:13.911434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43557 (local address 127.0.0.1:39116)
0620 00:11:13.911472 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:13.911482 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:11:13.911524 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:11:13.911670 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:11:13.911671 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:11:13.911834 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:11:13.911869 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:11:13.911871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.912709 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.912710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:11:13.913518 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:11:13.913521 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.914056 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.914058 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:11:13.914124 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:13.914444 (+ 320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:11:13.914449 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:11:13.916141 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:11:13.919097 (+ 2956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:11:13.919098 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:11:13.919099 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:11:13.919341 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:11:13.919560 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:11:13.919561 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:11:13.919561 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:11:13.919615 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:11:13.919928 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:11:13.919929 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:11:13.919993 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 00:11:13.920062 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 00:11:13.920159 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260620 00:11:13.920274 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:13.911475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39116 (local address 127.0.98.60:43557)
0620 00:11:13.911515 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:13.911517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:13.911525 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:11:13.911585 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:13.911586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:13.911594 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:13.911933 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:13.911956 (+ 23us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.912634 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.913621 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.913948 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.914051 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:13.916539 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:13.916540 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:13.916540 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:11:13.916548 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:13.919008 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:13.919433 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:13.919434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:13.919434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:13.919490 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:13.919685 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:13.919686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:13.919686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:13.919863 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:13.919924 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:13.920114 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 00:11:13.920189 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 00:11:13.922600 23058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 892e00cdd723432c9a76fe82dd40112f: Bootstrap starting.
I20260620 00:11:13.923386 23058 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 892e00cdd723432c9a76fe82dd40112f: Neither blocks nor log segments found. Creating new log.
I20260620 00:11:13.923705 23058 log.cc:824] T 00000000000000000000000000000000 P 892e00cdd723432c9a76fe82dd40112f: Log is configured to *not* fsync() on all Append() calls
Jun 20 00:11:13 dist-test-slave-grqt krb5kdc[22977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 00:11:13.925225 23058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 892e00cdd723432c9a76fe82dd40112f: No bootstrap required, opened a new log
Jun 20 00:11:13 dist-test-slave-grqt krb5kdc[22977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 00:11:13.927582 23058 raft_consensus.cc:348] T 00000000000000000000000000000000 P 892e00cdd723432c9a76fe82dd40112f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892e00cdd723432c9a76fe82dd40112f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32833 } } peers { permanent_uuid: "997f0e95727940b19036ed7058c27008" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40037 } } peers { permanent_uuid: "e884fa7572614f6eb70b2938a986c753" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } }
I20260620 00:11:13.927765 23058 raft_consensus.cc:729] T 00000000000000000000000000000000 P 892e00cdd723432c9a76fe82dd40112f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 892e00cdd723432c9a76fe82dd40112f, State: Initialized, Role: FOLLOWER
I20260620 00:11:13.927861 23058 consensus_queue.cc:260] T 00000000000000000000000000000000 P 892e00cdd723432c9a76fe82dd40112f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892e00cdd723432c9a76fe82dd40112f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32833 } } peers { permanent_uuid: "997f0e95727940b19036ed7058c27008" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40037 } } peers { permanent_uuid: "e884fa7572614f6eb70b2938a986c753" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } }
I20260620 00:11:13.928197 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:13.918776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43557 (local address 127.0.0.1:39122)
0620 00:11:13.918823 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:13.918833 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:11:13.918868 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:11:13.919130 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:11:13.919130 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:11:13.919267 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:11:13.919292 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:11:13.919293 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.921580 (+ 2287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.921581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:11:13.922074 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:11:13.922075 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.922462 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.922462 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:11:13.922510 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:13.922881 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:11:13.922885 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:11:13.924177 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:11:13.927059 (+ 2882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:11:13.927060 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:11:13.927061 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:11:13.927276 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:11:13.927474 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:11:13.927475 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:11:13.927476 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:11:13.927551 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:11:13.927900 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:11:13.927900 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:11:13.927938 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 00:11:13.928003 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 00:11:13.928113 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 00:11:13.928233 23058 sys_catalog.cc:564] T 00000000000000000000000000000000 P 892e00cdd723432c9a76fe82dd40112f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:11:13.928179 23215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 892e00cdd723432c9a76fe82dd40112f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892e00cdd723432c9a76fe82dd40112f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32833 } } peers { permanent_uuid: "997f0e95727940b19036ed7058c27008" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40037 } } peers { permanent_uuid: "e884fa7572614f6eb70b2938a986c753" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } } }
I20260620 00:11:13.928552 23215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 892e00cdd723432c9a76fe82dd40112f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:11:13.928577 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:13.918808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39122 (local address 127.0.98.60:43557)
0620 00:11:13.919002 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:13.919003 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:13.919009 (+ 6us) server_negotiation.cc:407] Connection header received
0620 00:11:13.919024 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:13.919024 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:13.919033 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:13.919126 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:13.920649 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.921508 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.922160 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.922381 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.922463 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:13.924263 (+ 1800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:13.924264 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:13.924265 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:11:13.924271 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:13.926973 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:13.927349 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:13.927350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:13.927350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:13.927403 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:13.927626 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:13.927627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:13.927628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:13.927807 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:13.927876 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:13.928386 (+ 510us) server_negotiation.cc:299] Negotiation successful
0620 00:11:13.928468 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":151,"threads_started":1}
I20260620 00:11:13.929606 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:13.921016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33644 (local address 127.0.98.61:40037)
0620 00:11:13.921054 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:13.921056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:13.921062 (+ 6us) server_negotiation.cc:407] Connection header received
0620 00:11:13.921097 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:13.921098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:13.921107 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:13.921149 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:13.921780 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.922465 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.923696 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.924012 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.924089 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:13.926678 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:13.926679 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:13.926679 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:11:13.926687 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:13.928369 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:13.928797 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:13.928798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:13.928799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:13.928852 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:13.929074 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:13.929075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:13.929075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:13.929186 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:13.929374 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:13.929434 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 00:11:13.929510 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 00:11:13.930174 23226 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 892e00cdd723432c9a76fe82dd40112f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260620 00:11:13.930176 23208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e884fa7572614f6eb70b2938a986c753: Bootstrap starting.
W20260620 00:11:13.930229 23226 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:11:13.930889 23208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e884fa7572614f6eb70b2938a986c753: Neither blocks nor log segments found. Creating new log.
I20260620 00:11:13.931195 23208 log.cc:824] T 00000000000000000000000000000000 P e884fa7572614f6eb70b2938a986c753: Log is configured to *not* fsync() on all Append() calls
I20260620 00:11:13.932041 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:11:13.932308 23208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e884fa7572614f6eb70b2938a986c753: No bootstrap required, opened a new log
I20260620 00:11:13.932930 23032 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "892e00cdd723432c9a76fe82dd40112f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54046
I20260620 00:11:13.933048 23032 raft_consensus.cc:482] T 00000000000000000000000000000000 P 892e00cdd723432c9a76fe82dd40112f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 00:11:13.933089 23032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 892e00cdd723432c9a76fe82dd40112f [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:11:13.934039 23032 raft_consensus.cc:504] T 00000000000000000000000000000000 P 892e00cdd723432c9a76fe82dd40112f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892e00cdd723432c9a76fe82dd40112f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32833 } } peers { permanent_uuid: "997f0e95727940b19036ed7058c27008" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40037 } } peers { permanent_uuid: "e884fa7572614f6eb70b2938a986c753" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } }
I20260620 00:11:13.934286 23032 leader_election.cc:288] T 00000000000000000000000000000000 P 892e00cdd723432c9a76fe82dd40112f [CANDIDATE]: Term 1 election: Requested vote from peers 997f0e95727940b19036ed7058c27008 (127.0.98.61:40037), e884fa7572614f6eb70b2938a986c753 (127.0.98.60:43557)
I20260620 00:11:13.931892 22990 master_runner.cc:186] Error getting master registration for 127.0.98.61:40037: OK, instance_id { permanent_uuid: "997f0e95727940b19036ed7058c27008" instance_seqno: 1781914273724012 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 00:11:13.934818 22990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 00:11:13.935690 23208 raft_consensus.cc:348] T 00000000000000000000000000000000 P e884fa7572614f6eb70b2938a986c753 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892e00cdd723432c9a76fe82dd40112f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32833 } } peers { permanent_uuid: "997f0e95727940b19036ed7058c27008" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40037 } } peers { permanent_uuid: "e884fa7572614f6eb70b2938a986c753" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } }
I20260620 00:11:13.935935 23208 raft_consensus.cc:729] T 00000000000000000000000000000000 P e884fa7572614f6eb70b2938a986c753 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e884fa7572614f6eb70b2938a986c753, State: Initialized, Role: FOLLOWER
I20260620 00:11:13.936038 23208 consensus_queue.cc:260] T 00000000000000000000000000000000 P e884fa7572614f6eb70b2938a986c753 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892e00cdd723432c9a76fe82dd40112f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32833 } } peers { permanent_uuid: "997f0e95727940b19036ed7058c27008" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40037 } } peers { permanent_uuid: "e884fa7572614f6eb70b2938a986c753" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } }
I20260620 00:11:13.936350 23208 sys_catalog.cc:564] T 00000000000000000000000000000000 P e884fa7572614f6eb70b2938a986c753 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:11:13.936371 23182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "892e00cdd723432c9a76fe82dd40112f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e884fa7572614f6eb70b2938a986c753"
I20260620 00:11:13.936493 23182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e884fa7572614f6eb70b2938a986c753 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:11:13.937333 23182 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e884fa7572614f6eb70b2938a986c753 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 892e00cdd723432c9a76fe82dd40112f in term 1.
I20260620 00:11:13.937601 23009 leader_election.cc:302] T 00000000000000000000000000000000 P 892e00cdd723432c9a76fe82dd40112f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 892e00cdd723432c9a76fe82dd40112f, e884fa7572614f6eb70b2938a986c753; no voters:
I20260620 00:11:13.938445 23215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 892e00cdd723432c9a76fe82dd40112f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:11:13.938624 23215 raft_consensus.cc:686] T 00000000000000000000000000000000 P 892e00cdd723432c9a76fe82dd40112f [term 1 LEADER]: Becoming Leader. State: Replica: 892e00cdd723432c9a76fe82dd40112f, State: Running, Role: LEADER
I20260620 00:11:13.938702 23215 consensus_queue.cc:237] T 00000000000000000000000000000000 P 892e00cdd723432c9a76fe82dd40112f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892e00cdd723432c9a76fe82dd40112f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32833 } } peers { permanent_uuid: "997f0e95727940b19036ed7058c27008" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40037 } } peers { permanent_uuid: "e884fa7572614f6eb70b2938a986c753" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } }
I20260620 00:11:13.939153 23215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 892e00cdd723432c9a76fe82dd40112f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 892e00cdd723432c9a76fe82dd40112f. Latest consensus state: current_term: 1 leader_uuid: "892e00cdd723432c9a76fe82dd40112f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892e00cdd723432c9a76fe82dd40112f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32833 } } peers { permanent_uuid: "997f0e95727940b19036ed7058c27008" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40037 } } peers { permanent_uuid: "e884fa7572614f6eb70b2938a986c753" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } } }
I20260620 00:11:13.939253 23215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 892e00cdd723432c9a76fe82dd40112f [sys.catalog]: This master's current role is: LEADER
I20260620 00:11:13.941196 23236 sys_catalog.cc:455] T 00000000000000000000000000000000 P e884fa7572614f6eb70b2938a986c753 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892e00cdd723432c9a76fe82dd40112f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32833 } } peers { permanent_uuid: "997f0e95727940b19036ed7058c27008" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40037 } } peers { permanent_uuid: "e884fa7572614f6eb70b2938a986c753" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } } }
I20260620 00:11:13.941315 23236 sys_catalog.cc:458] T 00000000000000000000000000000000 P e884fa7572614f6eb70b2938a986c753 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 00:11:13.942091 23247 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e884fa7572614f6eb70b2938a986c753: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:11:13.942148 23247 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:11:13.941660 23140 master_runner.cc:186] Error getting master registration for 127.0.98.61:40037: OK, instance_id { permanent_uuid: "997f0e95727940b19036ed7058c27008" instance_seqno: 1781914273724012 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 00:11:13.945358 23140 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 00:11:13.945760 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:13.935729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33650 (local address 127.0.98.61:40037)
0620 00:11:13.935775 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:13.935777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:13.936270 (+ 493us) server_negotiation.cc:407] Connection header received
0620 00:11:13.937544 (+ 1274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:13.937545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:13.937559 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:13.937593 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:13.938333 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.939035 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.940443 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.940751 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.940794 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:13.942413 (+ 1619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:13.942414 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:13.942415 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:11:13.942421 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:13.944036 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:13.944438 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:13.944439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:13.944440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:13.944496 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:13.944697 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:13.944698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:13.944698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:13.944818 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:13.945329 (+ 511us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:13.945587 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 00:11:13.945667 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 00:11:13.946676 23250 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:11:13.946925 23250 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:11:13.948027 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:13.935946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39130 (local address 127.0.98.60:43557)
0620 00:11:13.935984 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:13.935986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:13.940379 (+ 4393us) server_negotiation.cc:407] Connection header received
0620 00:11:13.940408 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:13.940409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:13.940420 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:13.940482 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:13.941234 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.941999 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.942968 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.943287 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.943330 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:13.944743 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:13.944745 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:13.944745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:11:13.944752 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:13.946754 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:13.947175 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:13.947176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:13.947177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:13.947229 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:13.947456 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:13.947457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:13.947457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:13.947612 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:13.947685 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:13.947897 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 00:11:13.947972 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 00:11:13.951125 23061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:13.935548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54058 (local address 127.0.98.62:32833)
0620 00:11:13.935603 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:13.935605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:13.935812 (+ 207us) server_negotiation.cc:407] Connection header received
0620 00:11:13.936146 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:13.936147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:13.936161 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:13.936209 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:13.938059 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.938788 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.946136 (+ 7348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.946422 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.946496 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:13.947910 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:13.947912 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:13.947912 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:11:13.947920 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:13.949705 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:13.950223 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:13.950224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:13.950225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:13.950284 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:13.950550 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:13.950551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:13.950551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:13.950686 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:13.950793 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:13.950949 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 00:11:13.951028 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 00:11:13.958868 23182 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e884fa7572614f6eb70b2938a986c753 [term 1 FOLLOWER]: Refusing update from remote peer 892e00cdd723432c9a76fe82dd40112f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:11:13.959052 23133 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } attempt: 3
I20260620 00:11:13.959947 23248 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 892e00cdd723432c9a76fe82dd40112f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e884fa7572614f6eb70b2938a986c753" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:11:13.963121 23236 sys_catalog.cc:455] T 00000000000000000000000000000000 P e884fa7572614f6eb70b2938a986c753 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 892e00cdd723432c9a76fe82dd40112f. Latest consensus state: current_term: 1 leader_uuid: "892e00cdd723432c9a76fe82dd40112f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892e00cdd723432c9a76fe82dd40112f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32833 } } peers { permanent_uuid: "997f0e95727940b19036ed7058c27008" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40037 } } peers { permanent_uuid: "e884fa7572614f6eb70b2938a986c753" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } } }
I20260620 00:11:13.963223 23236 sys_catalog.cc:458] T 00000000000000000000000000000000 P e884fa7572614f6eb70b2938a986c753 [sys.catalog]: This master's current role is: FOLLOWER
Jun 20 00:11:13 dist-test-slave-grqt krb5kdc[22977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 00:11:13.964879 23248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 892e00cdd723432c9a76fe82dd40112f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "892e00cdd723432c9a76fe82dd40112f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892e00cdd723432c9a76fe82dd40112f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32833 } } peers { permanent_uuid: "997f0e95727940b19036ed7058c27008" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40037 } } peers { permanent_uuid: "e884fa7572614f6eb70b2938a986c753" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } } }
I20260620 00:11:13.964972 23248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 892e00cdd723432c9a76fe82dd40112f [sys.catalog]: This master's current role is: LEADER
I20260620 00:11:13.965554 23236 sys_catalog.cc:455] T 00000000000000000000000000000000 P e884fa7572614f6eb70b2938a986c753 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "892e00cdd723432c9a76fe82dd40112f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892e00cdd723432c9a76fe82dd40112f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32833 } } peers { permanent_uuid: "997f0e95727940b19036ed7058c27008" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40037 } } peers { permanent_uuid: "e884fa7572614f6eb70b2938a986c753" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } } }
I20260620 00:11:13.965637 23236 sys_catalog.cc:458] T 00000000000000000000000000000000 P e884fa7572614f6eb70b2938a986c753 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:11:13.965713 23252 mvcc.cc:205] Tried to move back new op lower bound from 7298720866132164608 to 7298720866053828608. Current Snapshot: MvccSnapshot[applied={T|T < 7298720866132164608}]
I20260620 00:11:13.965866 23253 mvcc.cc:205] Tried to move back new op lower bound from 7298720866132164608 to 7298720866053828608. Current Snapshot: MvccSnapshot[applied={T|T < 7298720866132164608}]
I20260620 00:11:13.966639 23250 catalog_manager.cc:1347] Generated new cluster ID: 3306dd08db4344239ebf9ecc2568843a
I20260620 00:11:13.966686 23250 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:11:13.968611 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:13.959303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43557 (local address 127.0.0.1:39146)
0620 00:11:13.959347 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 00:11:13.959358 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:11:13.959394 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:11:13.959599 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:11:13.959600 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:11:13.959780 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:11:13.959831 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:11:13.959833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.961300 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.961301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:11:13.962251 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:11:13.962254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.962870 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.962871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:11:13.962930 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:13.963409 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:11:13.963414 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:11:13.965060 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:11:13.967447 (+ 2387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:11:13.967449 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:11:13.967449 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:11:13.967666 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:11:13.967907 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:11:13.967907 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:11:13.967908 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:11:13.967974 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:11:13.968288 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:11:13.968289 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:11:13.968345 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 00:11:13.968404 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 00:11:13.968507 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 00:11:13.969230 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:13.959351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39146 (local address 127.0.98.60:43557)
0620 00:11:13.959400 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:13.959403 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:13.959448 (+ 45us) server_negotiation.cc:407] Connection header received
0620 00:11:13.959475 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:13.959476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:13.959486 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:13.959542 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:13.959927 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.960713 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.962410 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:13.962695 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:13.962757 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:13.965153 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:11:13.965155 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:11:13.965155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:11:13.965162 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:11:13.967274 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:13.967759 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:13.967760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:13.967760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:13.967827 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:11:13.968051 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:11:13.968051 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:11:13.968052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:11:13.968191 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:11:13.968247 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:13.969071 (+ 824us) server_negotiation.cc:299] Negotiation successful
0620 00:11:13.969150 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 00:11:13.972113 23133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 997f0e95727940b19036ed7058c27008: Bootstrap starting.
I20260620 00:11:13.972831 23133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 997f0e95727940b19036ed7058c27008: Neither blocks nor log segments found. Creating new log.
I20260620 00:11:13.973134 23133 log.cc:824] T 00000000000000000000000000000000 P 997f0e95727940b19036ed7058c27008: Log is configured to *not* fsync() on all Append() calls
I20260620 00:11:13.974151 23133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 997f0e95727940b19036ed7058c27008: No bootstrap required, opened a new log
I20260620 00:11:13.976197 23133 raft_consensus.cc:348] T 00000000000000000000000000000000 P 997f0e95727940b19036ed7058c27008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892e00cdd723432c9a76fe82dd40112f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32833 } } peers { permanent_uuid: "997f0e95727940b19036ed7058c27008" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40037 } } peers { permanent_uuid: "e884fa7572614f6eb70b2938a986c753" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } }
I20260620 00:11:13.976323 23133 raft_consensus.cc:729] T 00000000000000000000000000000000 P 997f0e95727940b19036ed7058c27008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 997f0e95727940b19036ed7058c27008, State: Initialized, Role: FOLLOWER
I20260620 00:11:13.976395 23133 consensus_queue.cc:260] T 00000000000000000000000000000000 P 997f0e95727940b19036ed7058c27008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892e00cdd723432c9a76fe82dd40112f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32833 } } peers { permanent_uuid: "997f0e95727940b19036ed7058c27008" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40037 } } peers { permanent_uuid: "e884fa7572614f6eb70b2938a986c753" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } }
I20260620 00:11:13.976712 23133 sys_catalog.cc:564] T 00000000000000000000000000000000 P 997f0e95727940b19036ed7058c27008 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:11:13.977563 23106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 997f0e95727940b19036ed7058c27008 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:11:13.978452 23106 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 997f0e95727940b19036ed7058c27008 [term 1 FOLLOWER]: Refusing update from remote peer 892e00cdd723432c9a76fe82dd40112f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:11:13.978632 23107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "892e00cdd723432c9a76fe82dd40112f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "997f0e95727940b19036ed7058c27008"
I20260620 00:11:13.979359 23107 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 997f0e95727940b19036ed7058c27008 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 892e00cdd723432c9a76fe82dd40112f in term 1.
I20260620 00:11:13.979705 23215 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 892e00cdd723432c9a76fe82dd40112f [LEADER]: Connected to new peer: Peer: permanent_uuid: "997f0e95727940b19036ed7058c27008" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:11:13.981897 23258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 997f0e95727940b19036ed7058c27008 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "892e00cdd723432c9a76fe82dd40112f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892e00cdd723432c9a76fe82dd40112f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32833 } } peers { permanent_uuid: "997f0e95727940b19036ed7058c27008" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40037 } } peers { permanent_uuid: "e884fa7572614f6eb70b2938a986c753" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } } }
I20260620 00:11:13.982395 23258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 997f0e95727940b19036ed7058c27008 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:11:13.982465 23258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 997f0e95727940b19036ed7058c27008 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 892e00cdd723432c9a76fe82dd40112f. Latest consensus state: current_term: 1 leader_uuid: "892e00cdd723432c9a76fe82dd40112f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892e00cdd723432c9a76fe82dd40112f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32833 } } peers { permanent_uuid: "997f0e95727940b19036ed7058c27008" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40037 } } peers { permanent_uuid: "e884fa7572614f6eb70b2938a986c753" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } } }
I20260620 00:11:13.982530 23258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 997f0e95727940b19036ed7058c27008 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:11:13.982560 23258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 997f0e95727940b19036ed7058c27008 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "892e00cdd723432c9a76fe82dd40112f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892e00cdd723432c9a76fe82dd40112f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32833 } } peers { permanent_uuid: "997f0e95727940b19036ed7058c27008" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40037 } } peers { permanent_uuid: "e884fa7572614f6eb70b2938a986c753" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } } }
I20260620 00:11:13.982606 23258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 997f0e95727940b19036ed7058c27008 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 00:11:13.986037 23273 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 997f0e95727940b19036ed7058c27008: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:11:13.986140 23273 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:11:13.986604 23270 mvcc.cc:205] Tried to move back new op lower bound from 7298720866132164608 to 7298720866053828608. Current Snapshot: MvccSnapshot[applied={T|T < 7298720866132164608}]
I20260620 00:11:13.992543 23215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 892e00cdd723432c9a76fe82dd40112f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "892e00cdd723432c9a76fe82dd40112f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892e00cdd723432c9a76fe82dd40112f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32833 } } peers { permanent_uuid: "997f0e95727940b19036ed7058c27008" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40037 } } peers { permanent_uuid: "e884fa7572614f6eb70b2938a986c753" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } } }
I20260620 00:11:13.992656 23215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 892e00cdd723432c9a76fe82dd40112f [sys.catalog]: This master's current role is: LEADER
I20260620 00:11:13.995093 23250 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:11:13.995965 23250 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:11:14.001152 23250 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 892e00cdd723432c9a76fe82dd40112f: Generated new TSK 0
I20260620 00:11:14.001446 23250 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 00:11:14.037495 23213 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:11:14.036033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39150 (local address 127.0.98.60:43557)
0620 00:11:14.036089 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:14.036091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:14.036767 (+ 676us) server_negotiation.cc:407] Connection header received
0620 00:11:14.036911 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:14.036913 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:14.036924 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:14.036955 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:14.037416 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39150: BlockingRecv error: recv got EOF from 127.0.0.1:39150 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 00:11:14.038347 23061 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:11:14.036440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54064 (local address 127.0.98.62:32833)
0620 00:11:14.036488 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:14.036489 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:14.037875 (+ 1386us) server_negotiation.cc:407] Connection header received
0620 00:11:14.037970 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:14.037971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:14.037983 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:14.038012 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:14.038291 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54064: BlockingRecv error: recv got EOF from 127.0.0.1:54064 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 00:11:14.039400 23137 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:11:14.038786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33656 (local address 127.0.98.61:40037)
0620 00:11:14.038823 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:14.038825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:14.038932 (+ 107us) server_negotiation.cc:407] Connection header received
0620 00:11:14.039022 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:14.039023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:14.039032 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:14.039061 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:14.039343 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33656: BlockingRecv error: recv got EOF from 127.0.0.1:33656 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 00:11:14.041769 23137 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:11:14.040941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33666 (local address 127.0.98.61:40037)
0620 00:11:14.040976 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:14.040979 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:14.041220 (+ 241us) server_negotiation.cc:407] Connection header received
0620 00:11:14.041319 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:14.041320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:14.041338 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:14.041399 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:14.041725 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33666: BlockingRecv error: recv got EOF from 127.0.0.1:33666 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 00:11:14.044720 23137 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:11:14.044029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33670 (local address 127.0.98.61:40037)
0620 00:11:14.044059 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:14.044061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:14.044251 (+ 190us) server_negotiation.cc:407] Connection header received
0620 00:11:14.044341 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:14.044342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:14.044351 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:14.044378 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:14.044674 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33670: BlockingRecv error: recv got EOF from 127.0.0.1:33670 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 00:11:14.048552 23137 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:11:14.047869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33672 (local address 127.0.98.61:40037)
0620 00:11:14.047899 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:14.047900 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:14.048086 (+ 186us) server_negotiation.cc:407] Connection header received
0620 00:11:14.048177 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:14.048178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:14.048187 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:14.048214 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:14.048508 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33672: BlockingRecv error: recv got EOF from 127.0.0.1:33672 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260620 00:11:14.054603 23137 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:11:14.053838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33674 (local address 127.0.98.61:40037)
0620 00:11:14.053876 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:14.053877 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:14.054083 (+ 206us) server_negotiation.cc:407] Connection header received
0620 00:11:14.054178 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:14.054179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:14.054189 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:14.054218 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:14.054551 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33674: BlockingRecv error: recv got EOF from 127.0.0.1:33674 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 00:11:14.064711 23137 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:11:14.063904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33680 (local address 127.0.98.61:40037)
0620 00:11:14.063940 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:14.063942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:14.064165 (+ 223us) server_negotiation.cc:407] Connection header received
0620 00:11:14.064265 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:14.064266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:14.064276 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:14.064305 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:14.064662 (+ 357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33680: BlockingRecv error: recv got EOF from 127.0.0.1:33680 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 00:11:14.083137 23137 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:11:14.082212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33690 (local address 127.0.98.61:40037)
0620 00:11:14.082257 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:14.082260 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:14.082530 (+ 270us) server_negotiation.cc:407] Connection header received
0620 00:11:14.082640 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:14.082641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:14.082654 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:14.082684 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:14.083075 (+ 391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33690: BlockingRecv error: recv got EOF from 127.0.0.1:33690 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 00:11:14.117733 23137 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:11:14.116784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33692 (local address 127.0.98.61:40037)
0620 00:11:14.116833 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:14.116835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:14.117089 (+ 254us) server_negotiation.cc:407] Connection header received
0620 00:11:14.117194 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:14.117195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:14.117206 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:14.117236 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:14.117674 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33692: BlockingRecv error: recv got EOF from 127.0.0.1:33692 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260620 00:11:14.183822 23137 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:11:14.183211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33696 (local address 127.0.98.61:40037)
0620 00:11:14.183260 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:14.183262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:14.183342 (+ 80us) server_negotiation.cc:407] Connection header received
0620 00:11:14.183394 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:14.183395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:14.183405 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:14.183457 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:14.183774 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33696: BlockingRecv error: recv got EOF from 127.0.0.1:33696 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260620 00:11:14.313820 23137 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:11:14.313017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33702 (local address 127.0.98.61:40037)
0620 00:11:14.313109 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:14.313111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:14.313171 (+ 60us) server_negotiation.cc:407] Connection header received
0620 00:11:14.313238 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:14.313239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:14.313253 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:14.313314 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:14.313740 (+ 426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33702: BlockingRecv error: recv got EOF from 127.0.0.1:33702 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 00:11:14.571651 23137 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:11:14.571010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33714 (local address 127.0.98.61:40037)
0620 00:11:14.571084 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:14.571086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:14.571111 (+ 25us) server_negotiation.cc:407] Connection header received
0620 00:11:14.571166 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:14.571167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:14.571179 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:14.571250 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:11:14.571593 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33714: BlockingRecv error: recv got EOF from 127.0.0.1:33714 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 00:11:14.942559 23247 catalog_manager.cc:1259] Loaded cluster ID: 3306dd08db4344239ebf9ecc2568843a
I20260620 00:11:14.942638 23247 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e884fa7572614f6eb70b2938a986c753: loading cluster ID for follower catalog manager: success
I20260620 00:11:14.943426 23247 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e884fa7572614f6eb70b2938a986c753: acquiring CA information for follower catalog manager: success
I20260620 00:11:14.943671 23247 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e884fa7572614f6eb70b2938a986c753: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 00:11:14.986665 23273 catalog_manager.cc:1259] Loaded cluster ID: 3306dd08db4344239ebf9ecc2568843a
I20260620 00:11:14.986749 23273 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 997f0e95727940b19036ed7058c27008: loading cluster ID for follower catalog manager: success
I20260620 00:11:14.987619 23273 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 997f0e95727940b19036ed7058c27008: acquiring CA information for follower catalog manager: success
I20260620 00:11:14.987954 23273 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 997f0e95727940b19036ed7058c27008: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 00:11:15.087436 23340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:15.084838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33720 (local address 127.0.98.61:40037)
0620 00:11:15.084979 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:15.084981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:15.084992 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:11:15.085064 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:15.085065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:15.085129 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:15.085204 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:11:15.085337 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:15.085863 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:15.086677 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:15.086867 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:15.086917 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:15.087097 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:11:15.087213 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:11:15.087274 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:15.087350 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 00:11:15.087368 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
I20260620 00:11:15.091643 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:15.089357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39162 (local address 127.0.98.60:43557)
0620 00:11:15.089489 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:15.089491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:15.089498 (+ 7us) server_negotiation.cc:407] Connection header received
0620 00:11:15.089559 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:15.089560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:15.089593 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:15.089662 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:11:15.089748 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:15.090225 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:15.090932 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:15.091148 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:15.091212 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:15.091323 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:11:15.091423 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:11:15.091477 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:15.091508 (+ 31us) server_negotiation.cc:299] Negotiation successful
0620 00:11:15.091525 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":74,"threads_started":1}
I20260620 00:11:15.096251 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:15.093314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39170 (local address 127.0.98.60:43557)
0620 00:11:15.093382 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:15.093384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:15.093554 (+ 170us) server_negotiation.cc:407] Connection header received
0620 00:11:15.093684 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:15.093685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:15.093697 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:15.093741 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:11:15.093891 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:15.094653 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:15.095473 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:15.095768 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:15.095832 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:15.095970 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:11:15.096000 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:11:15.096053 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:15.096134 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 00:11:15.096157 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 00:11:15.096562 23340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:15.093402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33728 (local address 127.0.98.61:40037)
0620 00:11:15.093447 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:15.093449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:15.093553 (+ 104us) server_negotiation.cc:407] Connection header received
0620 00:11:15.093669 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:15.093670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:15.093681 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:15.093730 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:11:15.093951 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:15.094664 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:15.095689 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:15.096010 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:15.096083 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:15.096289 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:11:15.096320 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:11:15.096372 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:15.096464 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 00:11:15.096485 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 00:11:15.096617 23352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:11:15.093226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54080 (local address 127.0.98.62:32833)
0620 00:11:15.093403 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 00:11:15.093405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:11:15.093467 (+ 62us) server_negotiation.cc:407] Connection header received
0620 00:11:15.093553 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:11:15.093554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:11:15.093612 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:11:15.093714 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:11:15.093808 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:15.094362 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:15.095385 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:11:15.095682 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:11:15.095755 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:11:15.095959 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:11:15.096073 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:11:15.096132 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 00:11:15.096495 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 00:11:15.096520 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":58,"threads_started":1}
I20260620 00:11:15.103892 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 22990
I20260620 00:11:15.109516 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 23065
I20260620 00:11:15.114406 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8WXHPi/build/release/bin/kudu with pid 23140
2026-06-20T00:11:15Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2816 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5624 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (273405 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST